Skip to content

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

feat(moe): fetch locked-layer experts through pinned staging (#742)

Xiaoze Fanc5b5b7243a74
diff --git a/docs/cli.md b/docs/cli.md
index e109bde0..34a9ffaa 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -89,8 +89,11 @@ See [models.md](models.md#moe-strategies) for what each strategy does.
| `--moe-cache-split` | equal | How the pools share the slots: `equal` (in proportion to their layer counts), or `profile:<path>`: every pool keeps `top_k` slots and the rest go to the most used experts in the profile's per-layer expert counts. With `offload`, a pool smaller than the expert count lowers `--max-running-requests` to its slots / top_k |
| `--kv-reserve-tokens` | 8192 | KV token floor reserved before `--moe-cache-auto` fills experts |
| `--moe-cpu-threads` | physical cores | CPU worker threads for the cpu/hybrid executor |
-| `--moe-cpu-layers` | all on GPU | With `offload`: which MoE layers decode on CPU (`3,7,11`, a count, a fraction, or `auto`). `auto` is for Windows/WSL only, where CUDA pinned memory is capped; every value needs an expert format the CPU executor serves (bf16, fp8_block, nvfp4, mxfp4, GGUF on x86 CPUs) |
+| `--moe-cpu-layers` | all on GPU | With `offload`/`hybrid`: which MoE layers decode on CPU (`3,7,11`, a count, a fraction, or `auto`). `auto` is for hosts that cap CUDA pinned memory (Windows/WSL): it pins layers from the middle out until CUDA refuses one and OS-locks the rest (`FREETOKEN_PIN_BUDGET_GB` and FTW checkpoints lock from a size estimate instead), see [models.md](models.md#hosts-that-cap-pinned-memory-wsl); every value needs an expert format the CPU executor serves (bf16, fp8_block, nvfp4, mxfp4, GGUF on x86 CPUs) |
| `--moe-hybrid-max-fetch` | auto | With `hybrid`: max experts fetched over PCIe per layer per step; rest computed on CPU |
+| `--moe-locked-fetch` | auto | With `hybrid` and locked layers: share of a locked layer's misses fetched per step into its GPU slots through pinned staging. Auto takes p / (2p + c) from the `ft bench bw` row of the layer's kind (at most 1 per step without a row); 0 = locked layers decode on CPU only |
+| `--moe-locked-max-fetch` | 8 | Most experts a locked layer fetches per step; sizes the staging (that many rows of the widest expert, per bank) |
+| `--moe-collect-stats` | off | Log decode MoE stats with each decode log line: hit rate, and per layer-step the misses, experts fetched and experts computed on CPU, for pinned, locked and CPU-only layers |
| `--moe-prefill-hit-d2d` | off | Prefill: copy cache-hit experts device-side, stream only misses (CUDA >= 13) |
| `--disable-moe-prefill-overlap` | overlap on | Disable the two-buffer prefill copy overlap |
diff --git a/docs/models.md b/docs/models.md
index f6d99af9..a9b66a24 100644
--- a/docs/models.md
+++ b/docs/models.md
@@ -62,6 +62,28 @@ have no resident path and stay on `offload`. Offload-only flags
`--moe-prefill-hit-d2d`, `--disable-moe-prefill-overlap`) are ignored with a
warning; pass `--moe-strategy offload` to use them.
+### Hosts that cap pinned memory (WSL)
+
+WSL caps CUDA pinned memory near half of RAM, so the expert banks of a large
+model may not all pin. With `--moe-cpu-layers auto`, FreeToken loads the banks
+first, then pins them layer by layer from the middle out until CUDA refuses one;
+the remaining head and tail layers are OS-locked instead (resident, but with no
+device address). After a refusal under WDDM (WSL and native Windows) it
+unpins about 3 GiB more (`FREETOKEN_PIN_PROBE_MARGIN_GB`).
+The probe runs only where pinning is known to be capped (WDDM, or
+`FREETOKEN_SIMULATE_PIN_CAP_GB` for testing). FTW checkpoints, dummy weights
+and models the CPU executor cannot serve take the size estimate instead, as
+does `FREETOKEN_PIN_BUDGET_GB`, which fixes the budget: the locked layers are
+then picked from a size estimate before the load, and a refused pin stops the
+boot.
+
+Under `hybrid`, a locked layer keeps GPU slots of its own pools. Each decode
+step it fetches a share of its misses (`--moe-locked-fetch`, by default from the `ft bench bw` profile): CPU threads
+copy the experts into a small pinned staging buffer
+(`--moe-locked-max-fetch` rows per bank) and the GPU copies them into the
+slots. The CPU computes the remaining misses, as on any hybrid layer. Under
+`offload`, locked layers decode on the CPU.
+
### NVLink-C2C hosts (GH200 / GB200)
The host link is ~450 GB/s per direction instead of PCIe's ~32-64 GB/s. The
diff --git a/python/freetoken/engine/config.py b/python/freetoken/engine/config.py
index a6c47dd0..9593667e 100644
--- a/python/freetoken/engine/config.py
+++ b/python/freetoken/engine/config.py
@@ -74,6 +74,13 @@ class EngineConfig:
# misses so the PCIe fetch and the CPU compute finish together (perfect overlap);
# falls back to a fixed cap of 1 without a usable `ft bench bw` profile.
moe_hybrid_max_fetch: int = -1
+ # Hybrid with locked layers (banks the pin quota could not take, OS-locked instead): a locked layer keeps GPU
+ # slots of its own pools and fetches this share of each decode step's misses through pinned staging; the rest
+ # are computed on the CPU. -1 (default) = auto: per layer from its `ft bench bw` row (bench_profile.locked_fetch_fraction),
+ # a fixed cap of 1 without one. 0 = locked layers decode on the CPU only. moe_locked_max_fetch caps the fetches per
+ # (layer, step) and sizes the staging (that many rows of the widest expert, per bank).
+ moe_locked_fetch: float = -1.0
+ moe_locked_max_fetch: int = 8
cuda_graph_bs: List[int] | None = None
cuda_graph_max_bs: int | None = None
page_size: int = 1
@@ -104,6 +111,13 @@ class EngineConfig:
mm: MultimodalConfig = field(default_factory=MultimodalConfig)
def __post_init__(self):
+ if not (self.moe_locked_fetch == -1 or 0.0 <= self.moe_locked_fetch <= 1.0):
+ raise ValueError(f"moe_locked_fetch must be -1 (auto) or in [0, 1], got {self.moe_locked_fetch}")
+ if self.moe_locked_fetch > 0 and round(self.moe_locked_fetch * (1 << 16)) == 0:
+ # the ensure kernel takes the share in Q16: this one rounds to 0 and the layers would never fetch
+ raise ValueError(f"moe_locked_fetch {self.moe_locked_fetch} rounds to 0 in Q16; pass 0 to keep locked layers CPU-only")
+ if self.moe_locked_max_fetch < 1:
+ raise ValueError(f"moe_locked_max_fetch must be at least 1, got {self.moe_locked_max_fetch}")
if self.moe_backend is None:
return
if self.moe_strategy != "auto":
diff --git a/python/freetoken/engine/engine.py b/python/freetoken/engine/engine.py
index b93de55c..7982095a 100644
--- a/python/freetoken/engine/engine.py
+++ b/python/freetoken/engine/engine.py
@@ -22,8 +22,9 @@ from freetoken.mm.config import ENCODER_SECTIONS
from freetoken.models import create_model, load_weight
from freetoken.models.weight import ftw_lacks_vision
from freetoken.moe import is_offload_moe_strategy
-from freetoken.moe.expert_banks import attach_resident_banks, load_expert_banks
+from freetoken.moe.expert_banks import attach_resident_banks, load_expert_banks, pin_probe_enabled
from freetoken.moe.host_banks import PinFailed
+from freetoken.moe.locked_fetch import attach_locked_fetch, check_locked_fetch_config, locked_pool_tags, settle_banks, staging_reserve
from freetoken.moe.offload_cache import OffloadMoeCache, attach_offload_moe_cache, bank_layouts
from freetoken.moe.pool_policy import group_layers, make_pools, pool_of_layer, resolve_profile, split_slots
from freetoken.utils import align_ceil, init_logger, is_sm90_family, is_sm100_family, mem_GB, torch_dtype
@@ -685,56 +686,35 @@ class Engine:
"this model's experts come in several formats, which are served from per-format GPU slot pools; "
"they do not combine with --moe-strategy cpu or --moe-cpu-layers (use offload or hybrid)"
)
- cpu_layer_ids = _resolve_cpu_layers(config, num_moe_layers, reserved=self._host_tables_bytes, method=method)
- _check_pin_budget(config, reserved=self._host_tables_bytes, method=method, layer_methods=layer_methods)
- # the kernels were picked for model_config.decode_target; --moe-cpu-layers auto may still find that every bank fits the pin budget
- decode_target = config.model_config.decode_target
- if decode_target == "cpu" and not cpu_layer_ids:
- decode_target = "gpu"
- # split residency: where pinning is quota-capped (_pin_budget_bytes), pin only the GPU layers' banks and mlock the CPU layers'
- # uncapped hosts keep every bank pinned (CPU decode reads them the same; overlap prefill stays on)
- # not applied to plain --moe-strategy cpu; all-locked under a cap = --moe-strategy offload --moe-cpu-layers 1.0
- split_residency = (
- bool(cpu_layer_ids)
- and config.moe_strategy in ("offload", "hybrid")
- and _pin_budget_bytes(self._host_tables_bytes) is not None
- )
- if config.moe_strategy == "cpu" and not split_residency:
- # cpu mode pins every bank for the prefill double buffer; over the pin cap that dies in cudaHostRegister, so lock everything instead
- from freetoken.moe.expert_banks import bank_bytes_estimate, ftw_bank_bytes
-
- budget = _pin_budget_bytes(self._host_tables_bytes)
- bank_bytes = None
- if budget is not None:
- bank_bytes = ftw_bank_bytes(config.model_path) or bank_bytes_estimate(config.model_config, method)
- if bank_bytes and bank_bytes > budget:
- split_residency = True
- logger.info_rank0(
- f"--moe-strategy cpu: banks {bank_bytes / 2**30:.2f} GiB exceed the "
- f"pin budget; OS-locking all layers instead of pinning"
- )
- if split_residency and config.moe_prefill_overlap:
- # locked (unregistered) layers cannot feed the async pinned H2D double buffer; their prefill is a synchronous pageable copy via materialize
- logger.info_rank0(
- "--moe-cpu-layers split residency: disabling MoE prefill overlap "
- "(locked layers prefill via synchronous pageable copies)"
- )
- object.__setattr__(config, "moe_prefill_overlap", False)
# Fast path: an FTW checkpoint loads its repacked banks directly.
# Slow path: load_expert_banks auto-picks parallel vs serial baseline by
# expert-tensor granularity. Both pin-after-fill.
# --expert-load: serial/parallel force the read; auto (None) lets load_expert_banks
# pick (parallel for scattered experts, with a low-RAM fallback to serial).
expert_parallel = {"serial": False, "parallel": True}.get(config.expert_load, None)
- requested_residency = None
- if split_residency:
+ check_locked_fetch_config(config)
+ # --moe-cpu-layers auto without a manual pin budget: load unpinned, then pin layers until the driver refuses (settle_banks)
+ # a model the CPU executor cannot serve takes the estimate path, which rejects it before the load when layers must lock
+ probe = pin_probe_enabled(config, mixed_formats=layer_methods is not None) and _cpu_moe_executor_viable(config.model_config)
+ if probe:
from freetoken.moe.host_banks import HostResidency
- requested_residency = [
- HostResidency.LOCKED.value if i in cpu_layer_ids
- else HostResidency.PINNED.value
- for i in range(config.model_config.num_moe_layers)
- ]
+ requested_residency = [HostResidency.DEFERRED.value] * num_moe_layers
+ cpu_layer_ids, decode_target, split_residency = frozenset(), None, False
+ else:
+ reserved = self._host_tables_bytes + staging_reserve(config, lambda: _bank_bytes(config, method, layer_methods))
+ cpu_layer_ids = _resolve_cpu_layers(config, num_moe_layers, reserved=reserved, method=method)
+ _check_pin_budget(config, reserved=reserved, method=method, layer_methods=layer_methods)
+ decode_target, split_residency = self._residency_split(config, cpu_layer_ids, method, capped=_pin_budget_bytes(reserved) is not None)
+ requested_residency = None
+ if split_residency:
+ from freetoken.moe.host_banks import HostResidency
+
+ requested_residency = [
+ HostResidency.LOCKED.value if i in cpu_layer_ids
+ else HostResidency.PINNED.value
+ for i in range(config.model_config.num_moe_layers)
+ ]
# read before the expert load, so a bad profile fails in seconds
expert_counts = resolve_profile(
config.moe_cache_split, config.model_path,
@@ -754,14 +734,25 @@ class Engine:
)
except PinFailed as exc:
raise RuntimeError(f"{exc}; {_pin_hint(self._host_tables_bytes)}") from exc
+ banks, probed, staging = settle_banks(config, banks, probe=probe, has_locked=split_residency)
+ if probe:
+ cpu_layer_ids = probed
+ decode_target, split_residency = self._residency_split(config, cpu_layer_ids, method, capped=True)
from freetoken.engine.cache_budget import expert_bytes_per_slot, layer_slot_bytes
self._moe_layer_slot_bytes = layer_slot_bytes(banks.sources)
- self._moe_pool_groups = group_layers(config.moe_cache_scope, bank_layouts(banks.sources, banks.layer_tags), self._moe_layer_slot_bytes)
- if len(self._moe_pool_groups) > 1 and cpu_layer_ids:
+ # hybrid: locked layers keep GPU slots of their own pools and fetch part of their misses through the pinned staging (moe/locked_fetch.py)
+ locked_fetch = staging is not None
+ layer_tags = locked_pool_tags(banks.layer_tags, cpu_layer_ids, num_moe_layers) if locked_fetch else banks.layer_tags
+ self._moe_pool_groups = group_layers(config.moe_cache_scope, bank_layouts(banks.sources, layer_tags), self._moe_layer_slot_bytes)
+ if locked_fetch:
+ # a locked refill costs a staged copy: place the locked pools after the pinned ones, which hold the prefill staging when they are big enough
+ self._moe_pool_groups.sort(key=lambda layers: layers[0] in cpu_layer_ids)
+ if len(self._moe_pool_groups) > 1 and cpu_layer_ids and not locked_fetch:
raise ValueError(
"several MoE slot pools (--moe-cache-scope per-layer, or experts in several quant formats) need every MoE "
- "layer to decode from the GPU slot cache; they do not combine with --moe-strategy cpu or --moe-cpu-layers"
+ "layer to decode from the GPU slot cache; they combine with --moe-cpu-layers only for locked layers under "
+ "--moe-strategy hybrid unless --moe-locked-fetch is 0"
)
self._moe_expert_counts = expert_counts if len(self._moe_pool_groups) > 1 else None
if len(self._moe_pool_groups) == 1 and config.moe_cache_split != "equal":
@@ -852,7 +843,9 @@ class Engine:
)
# before set_bank_sources: the residency validation and the copy plan's skip of non-pinned layers key on the CPU-layer set
cache.cpu_layer_ids = cpu_layer_ids
- cache.set_bank_sources(banks.sources, layer_residency=banks.layer_residency, layer_tags=banks.layer_tags, row_tail_bytes=banks.row_tail_bytes)
+ if locked_fetch:
+ cache.locked_hybrid = cpu_layer_ids
+ cache.set_bank_sources(banks.sources, layer_residency=banks.layer_residency, layer_tags=layer_tags, row_tail_bytes=banks.row_tail_bytes)
cache.set_alphas(banks.gate_up_alpha, banks.down_alpha)
# Must be set before CUDA graph capture so the (device-side) accumulation ops are
# captured and re-run on every decode replay.
@@ -863,10 +856,49 @@ class Engine:
self._init_cpu_moe_executor(config, cache, layers)
if decode_target == "hybrid":
self._resolve_hybrid_fetch(config, cache)
+ if locked_fetch:
+ attach_locked_fetch(cache, config, staging, self._profile_layer_rows)
self.ctx.moe_offload_cache = cache
self.moe_offload_cache = cache
return cache
+ def _residency_split(self, config: EngineConfig, cpu_layer_ids, method, *, capped: bool):
+ """(decode_target, split_residency) for a CPU-layer set; turns prefill overlap off when some layers stay unpinned."""
+ # the kernels were picked for model_config.decode_target; --moe-cpu-layers auto may still find that every bank fits the pin budget
+ decode_target = config.model_config.decode_target
+ if decode_target == "cpu" and not cpu_layer_ids:
+ decode_target = "gpu"
+ # split residency: where pinning is quota-capped (_pin_budget_bytes), pin only the GPU layers' banks and mlock the CPU layers'
+ # uncapped hosts keep every bank pinned (CPU decode reads them the same; overlap prefill stays on)
+ # not applied to plain --moe-strategy cpu; all-locked under a cap = --moe-strategy offload --moe-cpu-layers 1.0
+ split_residency = (
+ bool(cpu_layer_ids)
+ and config.moe_strategy in ("offload", "hybrid")
+ and capped
+ )
+ if config.moe_strategy == "cpu" and not split_residency:
+ # cpu mode pins every bank for the prefill double buffer; over the pin cap that dies in cudaHostRegister, so lock everything instead
+ from freetoken.moe.expert_banks import bank_bytes_estimate, ftw_bank_bytes
+
+ budget = _pin_budget_bytes(self._host_tables_bytes)
+ bank_bytes = None
+ if budget is not None:
+ bank_bytes = ftw_bank_bytes(config.model_path) or bank_bytes_estimate(config.model_config, method)
+ if bank_bytes and bank_bytes > budget:
+ split_residency = True
+ logger.info_rank0(
+ f"--moe-strategy cpu: banks {bank_bytes / 2**30:.2f} GiB exceed the "
+ f"pin budget; OS-locking all layers instead of pinning"
+ )
+ if split_residency and config.moe_prefill_overlap:
+ # locked (unregistered) layers cannot feed the async pinned H2D double buffer; their prefill is a synchronous pageable copy via materialize
+ logger.info_rank0(
+ "--moe-cpu-layers split residency: disabling MoE prefill overlap "
+ "(locked layers prefill via synchronous pageable copies)"
+ )
+ object.__setattr__(config, "moe_prefill_overlap", False)
+ return decode_target, split_residency
+
def _moe_pool_bytes(self, pool_sizes: list[int]) -> int:
return sum(s * self._moe_layer_slot_bytes[g[0]] for g, s in zip(self._moe_pool_groups, pool_sizes))
@@ -889,26 +921,19 @@ class Engine:
"""
if config.moe_hybrid_max_fetch >= 0:
return # explicit fixed cap
- from freetoken.moe.bench_profile import fetch_fraction, layer_kind_key, load_layer_kinds
- from freetoken.moe.cpu_executor import _format_label
+ from freetoken.moe.bench_profile import fetch_fraction
cache.hybrid_max_fetch = 1
- methods = offload_layer_methods(self.model)
- if methods is None:
+ layer_rows = self._profile_layer_rows()
+ if layer_rows is None:
logger.warning_rank0("--moe-hybrid-max-fetch auto: the MoE layers bind no expert method to look up in the `ft bench bw` profile; using a fixed fetch cap of 1")
return
- gpu_name, gpu_uuid = _profile_gpu(self.device.index)
- rows = load_layer_kinds(gpu_name=gpu_name, gpu_uuid=gpu_uuid) or {}
executor = self.cpu_moe_executor
- keys, labels = [], []
- for m in methods:
- fmt, opts = m.cpu_spec()
- c = m.cfg
- keys.append(layer_kind_key(fmt, opts, c.hidden, c.local_intermediate, c.top_k, executor.num_threads))
- labels.append(f"{_format_label(fmt, tuple(sorted(opts.items())))} {c.hidden}x{c.local_intermediate}")
+ keys, labels, rows = layer_rows
fractions = [fetch_fraction(rows.get(key)) for key in keys]
cache.hybrid_fetch_fraction = [f or 0.0 for f in fractions]
- groups = collections.Counter(zip(labels, fractions))
+ # locked layers fetch the fetch_plan share instead; their own log line lists it
+ groups = collections.Counter((labels[l], fractions[l]) for l in range(len(keys)) if l not in cache.locked_hybrid)
fetched = [f"{f:.1%} on {n} {label} layers" for (label, f), n in groups.items() if f is not None]
if fetched:
logger.info_rank0(
@@ -928,6 +953,24 @@ class Engine:
f"{executor.isa}; using them anyway"
)
+ def _profile_layer_rows(self):
+ """(row key, label) of each MoE layer's kind in the `ft bench bw` profile, and the profile's rows; None without expert methods."""
+ from freetoken.moe.bench_profile import layer_kind_key, load_layer_kinds
+ from freetoken.moe.cpu_executor import _format_label
+
+ methods = offload_layer_methods(self.model)
+ if methods is None:
+ return None
+ gpu_name, gpu_uuid = _profile_gpu(self.device.index)
+ rows = load_layer_kinds(gpu_name=gpu_name, gpu_uuid=gpu_uuid) or {}
+ keys, labels = [], []
+ for m in methods:
+ fmt, opts = m.cpu_spec()
+ c = m.cfg
+ keys.append(layer_kind_key(fmt, opts, c.hidden, c.local_intermediate, c.top_k, self.cpu_moe_executor.num_threads))
+ labels.append(f"{_format_label(fmt, tuple(sorted(opts.items())))} {c.hidden}x{c.local_intermediate}")
+ return keys, labels, rows
+
def _init_cpu_moe_executor(self, config: EngineConfig, cache, layers) -> None:
"""Build the persistent CPU MoE executor (decode-time expert compute).
@@ -1587,6 +1630,8 @@ _DENSE_MOE_SETTINGS = {
"moe_cpu_layers": None,
"moe_cpu_threads": 0,
"moe_hybrid_max_fetch": -1,
+ "moe_locked_fetch": -1.0,
+ "moe_locked_max_fetch": 8,
"moe_prefill_overlap": True,
"moe_prefill_hit_d2d": False,
"expert_load": "auto",
diff --git a/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp b/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp
index 8162138c..b0de7085 100644
--- a/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp
+++ b/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp
@@ -28,17 +28,10 @@
#include <freetoken/hip_compat.h>
#include <torch/extension.h>
-#if defined(__linux__)
-#include <pthread.h>
-#include <sched.h>
-#define CPU_MOE_HAS_AFFINITY 1
-#else
-#define CPU_MOE_HAS_AFFINITY 0
-#endif
-
#include "format.h"
#include "gpu_sync.h"
#include "gpu_sync_io.h"
+#include "threads.h"
namespace freetoken::cpu_moe {
@@ -50,28 +43,10 @@ std::unique_ptr<MoeFormat> make_fp8_block_format(const MoeParams& p, const Forma
std::unique_ptr<MoeFormat> make_gguf_format(const MoeParams& p, const FormatArgs& a);
std::unique_ptr<MoeFormat> make_gguf_scaled_format(const MoeParams& p, const FormatArgs& a);
void set_ggml_cpu(uintptr_t addr);
+void bind_locked_fetch(pybind11::module_& m);
namespace {
-inline void pin_this_thread(int cpu) {
-#if CPU_MOE_HAS_AFFINITY
- cpu_set_t set;
- CPU_ZERO(&set);
- CPU_SET(cpu, &set);
- pthread_setaffinity_np(pthread_self(), sizeof(set), &set);
-#else
- (void)cpu;
-#endif
-}
-
-inline void cpu_relax() {
-#if CPU_MOE_X86
- _mm_pause();
-#elif defined(__aarch64__)
- asm volatile("yield" ::: "memory");
-#endif
-}
-
constexpr int kMaxTopK = 64;
// One MoE layer on the first num_tokens rows of the pinned IO.
@@ -566,6 +541,7 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
[](CpuMoeExecutor& e, bool v) { e.spin_wait.store(v, std::memory_order_relaxed); },
py::arg("enabled"))
.def("get_spin_wait", [](const CpuMoeExecutor& e) { return e.spin_wait.load(); });
+ freetoken::cpu_moe::bind_locked_fetch(m);
m.def("memops_probe", &GpuSync::probe, py::arg("stream"));
m.def("kernel_io_probe", &freetoken::cpu_moe::KernelIo::probe, py::arg("stream"));
m.def("host_func_capture_safe", &GpuSync::host_func_capture_safe);
diff --git a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync.h b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync.h
index 829a8f34..0e447751 100644
--- a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync.h
+++ b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync.h
@@ -44,8 +44,9 @@ inline void flag_store_release(volatile int64_t* p, int64_t v) {
class GpuSync {
public:
- // Allocates the handshake memory; throws if this runtime cannot.
- GpuSync();
+ // Allocates the handshake memory; throws if this runtime cannot, or lacks stream memops while require_memops.
+ // Without memops only the words, cpu_poll and cpu_done serve: kernels write the words and wait on them.
+ explicit GpuSync(bool require_memops = true);
~GpuSync();
GpuSync(const GpuSync&) = delete;
GpuSync& operator=(const GpuSync&) = delete;
diff --git a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_cuda.cpp b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_cuda.cpp
index a2a089bf..728776d3 100644
--- a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_cuda.cpp
+++ b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_cuda.cpp
@@ -94,8 +94,8 @@ struct GpuSync::Impl {
volatile int64_t* done = nullptr;
};
-GpuSync::GpuSync() : impl_(std::make_unique<Impl>()) {
- if (!cumemop_resolve()) throw std::runtime_error("CPU MoE flag sync: CUDA stream memops are unavailable");
+GpuSync::GpuSync(bool require_memops) : impl_(std::make_unique<Impl>()) {
+ if (require_memops && !cumemop_resolve()) throw std::runtime_error("CPU MoE flag sync: CUDA stream memops are unavailable");
const cudaError_t rc = cudaHostAlloc(&impl_->block, 128, cudaHostAllocPortable | cudaHostAllocMapped);
if (rc != cudaSuccess)
throw std::runtime_error(std::string("CPU MoE flag allocation failed: ") + cudaGetErrorString(rc));
diff --git a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_hip.cpp b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_hip.cpp
index 31ff9be5..395f2684 100644
--- a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_hip.cpp
+++ b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_hip.cpp
@@ -162,7 +162,8 @@ struct GpuSync::Impl {
hipStreamBatchMemOpParams wait_op{};
};
-GpuSync::GpuSync() : impl_(std::make_unique<Impl>()) {
+// The memop-free mode serves only the locked fetch, which ROCm keeps off: every GpuSync here needs memops.
+GpuSync::GpuSync(bool) : impl_(std::make_unique<Impl>()) {
hipError_t rc = alloc_signal(&impl_->ready);
if (rc == hipSuccess) rc = alloc_signal(&impl_->done);
if (rc != hipSuccess) {
diff --git a/python/freetoken/kernel/csrc/cpu_moe/locked_fetch.cpp b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch.cpp
new file mode 100644
index 00000000..e2e7a895
--- /dev/null
+++ b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch.cpp
@@ -0,0 +1,205 @@
+// Locked-layer fetch for hybrid decode (python/freetoken/moe/locked_fetch.py).
+//
+// A locked layer's host banks have no device address, so the GPU cannot gather its rows. Per decode step a publish kernel
+// writes the fetch list (count + layer-local expert rows) into mapped meta and rings a doorbell (GpuSync's words); the
+// coordinator and worker threads memcpy each listed row of every bank into pinned staging, then complete the doorbell,
+// and fast_index_copy_multi copies staging row i into the i-th evicted slot after a stream memop wait, or with its own
+// spin wait where memops do not work. Everything is kernels and memops, so it captures into the decode graph. Row i of a
+// bank sits at i * the layer's own row bytes, so layers of different widths share staging.
+
+#include <algorithm>
+#include <atomic>
+#include <chrono>
+#include <cstdint>
+#include <cstring>
+#include <memory>
+#include <stdexcept>
+#include <thread>
+#include <vector>
+
+#include <torch/extension.h>
+
+#include <freetoken/hip_compat.h>
+
+#include "gpu_sync.h"
+#include "locked_fetch_io.h"
+#include "threads.h"
+
+namespace freetoken::cpu_moe {
+namespace {
+
+using fetch_clock = std::chrono::steady_clock;
+// same idle policy as the MoE coordinator: spin while decode traffic flows, sleep once it stops
+constexpr auto kHotWindow = std::chrono::milliseconds(50);
+constexpr int64_t kChunkBytes = 256 << 10;
+
+struct CopyJob {
+ char* dst;
+ const char* src;
+ int64_t bytes;
+};
+
+class LockedFetch {
+ public:
+ // src[l][b] / row_bytes[l][b]: host base and row bytes of layer l's bank b (0 for banks or layers it never fetches).
+ // memops: wait on the doorbell with a stream memop, else the gather spins on done_word().
+ LockedFetch(std::vector<std::vector<uintptr_t>> src, std::vector<std::vector<int64_t>> row_bytes, std::vector<uintptr_t> staging,
+ int max_rows, std::vector<int> cpus, bool memops)
+ : src_(std::move(src)), row_bytes_(std::move(row_bytes)), staging_(std::move(staging)), max_rows_(max_rows), memops_(memops) {
+ sync_ = std::make_unique<GpuSync>(memops);
+ void* meta = nullptr;
+ void* meta_dev = nullptr;
+ const size_t meta_bytes = sizeof(int64_t) * (1 + (max_rows + 1) / 2);
+ if (cudaHostAlloc(&meta, meta_bytes, cudaHostAllocPortable | cudaHostAllocMapped) != cudaSuccess ||
+ cudaHostGetDevicePointer(&meta_dev, meta, 0) != cudaSuccess) {
+ (void)cudaGetLastError();
+ if (meta) (void)cudaFreeHost(meta);
+ throw std::runtime_error("locked fetch: mapped meta allocation failed");
+ }
+ meta_ = meta;
+ count_ = static_cast<volatile int64_t*>(meta);
+ rows_ = reinterpret_cast<const int32_t*>(static_cast<int64_t*>(meta) + 1);
+ count_dev_ = static_cast<int64_t*>(meta_dev);
+ rows_dev_ = reinterpret_cast<int32_t*>(static_cast<int64_t*>(meta_dev) + 1);
+ const int coord_cpu = cpus.empty() ? -1 : cpus[0];
+ for (size_t i = 1; i < cpus.size(); ++i) workers_.emplace_back([this, cpu = cpus[i]] { worker_loop(cpu); });
+ coord_ = std::thread([this, coord_cpu] { coordinator_loop(coord_cpu); });
+ }
+
+ ~LockedFetch() {
+ stop_.store(true);
+ coord_.join();
+ for (auto& t : workers_) t.join();
+ (void)cudaFreeHost(meta_);
+ }
+
+ LockedFetch(const LockedFetch&) = delete;
+ LockedFetch& operator=(const LockedFetch&) = delete;
+
+ // Hands one layer step's fetch list (the hybrid ensure's num_indices and src_indices) to the copy threads, then waits for
+ // them on `stream` when memops work; otherwise the gather that follows waits on done_word() itself.
+ void publish(uintptr_t stream, int layer, uintptr_t num_indices, uintptr_t src_indices) {
+ locked_publish(stream, reinterpret_cast<const int64_t*>(num_indices), reinterpret_cast<const int32_t*>(src_indices), max_rows_,
+ count_dev_, rows_dev_, reinterpret_cast<volatile int64_t*>(sync_->ready_word()),
+ reinterpret_cast<volatile int64_t*>(sync_->done_word()), static_cast<int64_t>(layer) + 1);
+ if (memops_) sync_->gpu_wait(stream);
+ }
+ uintptr_t done_word() const { return sync_->done_word(); }
+
+ private:
+ void run_jobs() {
+ for (;;) {
+ const int64_t i = next_.fetch_add(1, std::memory_order_relaxed);
+ if (i >= static_cast<int64_t>(jobs_.size())) return;
+ std::memcpy(jobs_[i].dst, jobs_[i].src, static_cast<size_t>(jobs_[i].bytes));
+ }
+ }
+
+ void worker_loop(int cpu) {
+ if (cpu >= 0) pin_this_thread(cpu);
+ uint64_t seen = 0;
+ auto last_active = fetch_clock::now();
+ int64_t sleep_us = 20;
+ while (!stop_.load(std::memory_order_relaxed)) {
+ const uint64_t gen = gen_.load(std::memory_order_acquire);
+ if (gen == seen) {
+ if (fetch_clock::now() - last_active < kHotWindow) {
+ cpu_relax();
+ } else {
+ std::this_thread::sleep_for(std::chrono::microseconds(sleep_us));
+ sleep_us = std::min<int64_t>(sleep_us * 2, 2000);
+ }
+ continue;
+ }
+ seen = gen;
+ run_jobs();
+ finished_.fetch_add(1, std::memory_order_acq_rel);
+ last_active = fetch_clock::now();
+ sleep_us = 20;
+ }
+ }
+
+ // Splits this step's rows into chunk jobs and runs them with the workers.
+ void copy_layer(int layer) {
+ int64_t n = *count_;
+ n = std::max<int64_t>(0, std::min<int64_t>(n, max_rows_));
+ jobs_.clear();
+ const auto& src = src_[layer];
+ const auto& row_bytes = row_bytes_[layer];
+ for (int64_t i = 0; i < n; ++i) {
+ const int64_t row = rows_[i];
+ for (size_t b = 0; b < staging_.size(); ++b) {
+ const int64_t rb = row_bytes[b];
+ if (rb == 0) continue;
+ char* dst = reinterpret_cast<char*>(staging_[b]) + i * rb;
+ const char* s = reinterpret_cast<const char*>(src[b]) + row * rb;
+ for (int64_t off = 0; off < rb; off += kChunkBytes) jobs_.push_back({dst + off, s + off, std::min(kChunkBytes, rb - off)});
+ }
+ }
+ if (jobs_.empty()) return;
+ next_.store(0, std::memory_order_relaxed);
+ finished_.store(0, std::memory_order_relaxed);
+ gen_.fetch_add(1, std::memory_order_acq_rel);
+ run_jobs();
+ // on stop a worker may leave before it joins this generation; the coordinator's run_jobs already took every job
+ while (finished_.load(std::memory_order_acquire) < static_cast<int64_t>(workers_.size()) && !stop_.load(std::memory_order_relaxed))
+ cpu_relax();
+ }
+
+ void coordinator_loop(int cpu) {
+ if (cpu >= 0) pin_this_thread(cpu);
+ auto last_active = fetch_clock::now();
+ int64_t sleep_us = 20;
+ while (!stop_.load(std::memory_order_relaxed)) {
+ const uint64_t task = sync_->cpu_poll();
+ if (task == 0) {
+ if (fetch_clock::now() - last_active < kHotWindow) {
+ cpu_relax();
+ } else {
+ std::this_thread::sleep_for(std::chrono::microseconds(sleep_us));
+ sleep_us = std::min<int64_t>(sleep_us * 2, 2000);
+ }
+ continue;
+ }
+ copy_layer(static_cast<int>(task - 1));
+ // release: the staging stores are visible before the GPU sees done
+ sync_->cpu_done();
+ last_active = fetch_clock::now();
+ sleep_us = 20;
+ }
+ sync_->cpu_done(); // a replay caught mid-teardown does not stall
+ }
+
+ std::vector<std::vector<uintptr_t>> src_;
+ std::vector<std::vector<int64_t>> row_bytes_;
+ std::vector<uintptr_t> staging_;
+ int64_t max_rows_;
+ bool memops_;
+ std::unique_ptr<GpuSync> sync_;
+ void* meta_ = nullptr;
+ const volatile int64_t* count_ = nullptr;
+ const int32_t* rows_ = nullptr;
+ int64_t* count_dev_ = nullptr;
+ int32_t* rows_dev_ = nullptr;
+ std::vector<CopyJob> jobs_;
+ std::atomic<int64_t> next_{0};
+ std::atomic<int64_t> finished_{0};
+ std::atomic<uint64_t> gen_{0};
+ std::atomic<bool> stop_{false};
+ std::vector<std::thread> workers_;
+ std::thread coord_;
+};
+
+} // namespace
+
+void bind_locked_fetch(pybind11::module_& m) {
+ namespace py = pybind11;
+ py::class_<LockedFetch>(m, "LockedFetch")
+ .def(py::init<std::vector<std::vector<uintptr_t>>, std::vector<std::vector<int64_t>>, std::vector<uintptr_t>, int, std::vector<int>,
+ bool>(),
+ py::arg("src"), py::arg("row_bytes"), py::arg("staging"), py::arg("max_rows"), py::arg("cpus"), py::arg("memops"))
+ .def("publish", &LockedFetch::publish, py::arg("stream"), py::arg("layer"), py::arg("num_indices"), py::arg("src_indices"))
+ .def("done_word", &LockedFetch::done_word);
+}
+
+} // namespace freetoken::cpu_moe
diff --git a/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.cu b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.cu
new file mode 100644
index 00000000..a264a549
--- /dev/null
+++ b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.cu
@@ -0,0 +1,52 @@
+#include <stdexcept>
+#include <string>
+
+#include <freetoken/hip_compat.h>
+
+#include "locked_fetch_io.h"
+
+namespace freetoken::cpu_moe {
+namespace {
+
+__device__ inline int64_t clamp_rows(const int64_t* num_indices, int max_rows) {
+ const int64_t n = *num_indices;
+ return n < 0 ? 0 : (n > max_rows ? max_rows : n);
+}
+
+// One block. The rows land before the doorbell, which the CPU reads with acquire.
+__global__ void publish_kernel(const int64_t* num_indices, const int32_t* src_indices, int max_rows, int64_t* meta_count,
+ int32_t* meta_rows, volatile int64_t* ready, volatile int64_t* done, int64_t task) {
+ const int64_t n = clamp_rows(num_indices, max_rows);
+ if (n == 0) {
+ if (threadIdx.x == 0) {
+ *done = 1;
+ __threadfence_system();
+ }
+ return;
+ }
+ for (int64_t i = threadIdx.x; i < n; i += blockDim.x) meta_rows[i] = src_indices[i];
+ if (threadIdx.x == 0) *meta_count = n;
+ __threadfence_system();
+ __syncthreads();
+ if (threadIdx.x == 0) {
+ *done = 0;
+ __threadfence_system();
+ *ready = task;
+ __threadfence_system();
+ }
+}
+
+void check(cudaError_t rc, const char* what) {
+ if (rc != cudaSuccess) throw std::runtime_error(std::string("locked fetch: ") + what + ": " + cudaGetErrorString(rc));
+}
+
+} // namespace
+
+void locked_publish(uintptr_t stream, const int64_t* num_indices, const int32_t* src_indices, int max_rows, int64_t* meta_count,
+ int32_t* meta_rows, volatile int64_t* ready, volatile int64_t* done, int64_t task) {
+ publish_kernel<<<1, 256, 0, reinterpret_cast<cudaStream_t>(stream)>>>(num_indices, src_indices, max_rows, meta_count, meta_rows, ready,
+ done, task);
+ check(cudaGetLastError(), "publish launch");
+}
+
+} // namespace freetoken::cpu_moe
diff --git a/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.h b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.h
new file mode 100644
index 00000000..57e6e2ef
--- /dev/null
+++ b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.h
@@ -0,0 +1,14 @@
+// GPU side of the locked-layer fetch (locked_fetch.cpp): one publish kernel in place of the meta copies and the doorbell writes.
+// One source for both runtimes like gpu_sync_io.cu.
+#pragma once
+
+#include <cstdint>
+
+namespace freetoken::cpu_moe {
+
+// Writes min(*num_indices, max_rows) and that many src rows into the mapped meta, then done = 0 and ready = task.
+// A step with nothing to fetch writes done = 1 and rings nothing, so the wait and the gather pass at once.
+void locked_publish(uintptr_t stream, const int64_t* num_indices, const int32_t* src_indices, int max_rows, int64_t* meta_count,
+ int32_t* meta_rows, volatile int64_t* ready, volatile int64_t* done, int64_t task);
+
+} // namespace freetoken::cpu_moe
diff --git a/python/freetoken/kernel/csrc/cpu_moe/threads.h b/python/freetoken/kernel/csrc/cpu_moe/threads.h
new file mode 100644
index 00000000..f7d9306b
--- /dev/null
+++ b/python/freetoken/kernel/csrc/cpu_moe/threads.h
@@ -0,0 +1,35 @@
+// Thread placement and spin helpers shared by the CPU MoE pool and the locked-layer fetch threads.
+#pragma once
+
+#include "common.h"
+
+#if defined(__linux__)
+#include <pthread.h>
+#include <sched.h>
+#define CPU_MOE_HAS_AFFINITY 1
+#else
+#define CPU_MOE_HAS_AFFINITY 0
+#endif
+
+namespace freetoken::cpu_moe {
+
+inline void pin_this_thread(int cpu) {
+#if CPU_MOE_HAS_AFFINITY
+ cpu_set_t set;
+ CPU_ZERO(&set);
+ CPU_SET(cpu, &set);
+ pthread_setaffinity_np(pthread_self(), sizeof(set), &set);
+#else
+ (void)cpu;
+#endif
+}
+
+inline void cpu_relax() {
+#if CPU_MOE_X86
+ _mm_pause();
+#elif defined(__aarch64__)
+ asm volatile("yield" ::: "memory");
+#endif
+}
+
+} // namespace freetoken::cpu_moe
diff --git a/python/freetoken/kernel/csrc/include/freetoken/hip_compat.h b/python/freetoken/kernel/csrc/include/freetoken/hip_compat.h
index 8abcd1a0..d1a22fe4 100644
--- a/python/freetoken/kernel/csrc/include/freetoken/hip_compat.h
+++ b/python/freetoken/kernel/csrc/include/freetoken/hip_compat.h
@@ -72,6 +72,10 @@
#define cudaHostRegister hipHostRegister
#endif
+#ifndef cudaHostUnregister
+#define cudaHostUnregister hipHostUnregister
+#endif
+
#ifndef cudaHostRegisterPortable
#define cudaHostRegisterPortable hipHostRegisterPortable
#endif
diff --git a/python/freetoken/kernel/csrc/jit/fast_index_copy.cuh b/python/freetoken/kernel/csrc/jit/fast_index_copy.cuh
index 43f3fc1b..279b15ae 100644
--- a/python/freetoken/kernel/csrc/jit/fast_index_copy.cuh
+++ b/python/freetoken/kernel/csrc/jit/fast_index_copy.cuh
@@ -507,13 +507,15 @@ struct FastIndexCopyKernel {
// block b copies bank `b = blockIdx.x / kBlocksPerBank`, grid-striding over that
// bank's (num_indices * feat/16) 16-byte units. Pointers + feature sizes are passed
// as small device arrays (built once by the cache), so it stays CUDA-graph capturable.
+// With wait_flag set, each block first waits for *wait_flag >= 1, a flag in mapped host memory the CPU sets once the source rows are written.
struct MultiIndexCopyParams {
const int64_t* __restrict__ dst_ptrs; // [B] device, each base addr of a bank slot cache
const int64_t* __restrict__ src_ptrs; // [B] device, each GPU-visible base addr of a bank host source
const int64_t* __restrict__ feat_bytes; // [B] device, per-row bytes (multiple of 16)
const void* __restrict__ dst_indices; // [L]
- const void* __restrict__ src_indices; // [L]
+ const void* __restrict__ src_indices; // [L], or null: row i copies source row i
const int64_t* __restrict__ valid_length; // [1] or null
+ const volatile int64_t* wait_flag; // GPU-visible address, or null: no wait
int64_t length; // max L
int num_banks;
};
@@ -522,6 +524,18 @@ template <typename IdType, std::size_t kNumThreads, std::size_t kBlocksPerBank>
__global__ __launch_bounds__(kNumThreads) void fast_index_copy_multi(
const __grid_constant__ MultiIndexCopyParams p
) {
+ if (p.wait_flag != nullptr) {
+ if (threadIdx.x == 0) {
+ while (*p.wait_flag < 1) {
+#if !FREETOKEN_USE_ROCM && __CUDA_ARCH__ >= 700
+ __nanosleep(100);
+#endif
+ }
+ // with the volatile read above, an acquire: the block reads the rows written before the flag
+ __threadfence_system();
+ }
+ __syncthreads();
+ }
const int b = static_cast<int>(blockIdx.x / kBlocksPerBank);
if (b >= p.num_banks) {
return;
@@ -540,7 +554,7 @@ __global__ __launch_bounds__(kNumThreads) void fast_index_copy_multi(
const int64_t row = u / units;
const int64_t col = (u - row * units) << 4; // byte offset within the row
const int64_t pd = static_cast<int64_t>(di[row]);
- const int64_t ps = static_cast<int64_t>(si[row]);
+ const int64_t ps = si ? static_cast<int64_t>(si[row]) : row;
const uint4 v = *reinterpret_cast<const uint4*>(src + ps * feat + col);
*reinterpret_cast<uint4*>(dst + pd * feat + col) = v;
}
@@ -553,8 +567,9 @@ struct MultiIndexCopyKernel {
tvm::ffi::TensorView src_ptrs,
tvm::ffi::TensorView feat_bytes,
tvm::ffi::TensorView dst_indices,
- tvm::ffi::TensorView src_indices,
- tvm::ffi::Optional<tvm::ffi::TensorView> num_indices
+ tvm::ffi::Optional<tvm::ffi::TensorView> src_indices,
+ tvm::ffi::Optional<tvm::ffi::TensorView> num_indices,
+ int64_t wait_flag
) {
using namespace host;
auto device = SymbolicDevice{};
@@ -567,7 +582,13 @@ struct MultiIndexCopyKernel {
TensorMatcher({B}).with_dtype<int64_t>(ptr_dtype).with_device<kDLCUDA, kDLROCM>(device)
.verify(dst_ptrs).verify(src_ptrs).verify(feat_bytes);
TensorMatcher({L}).with_dtype<int32_t, int64_t>(indices_dtype).with_device<kDLCUDA, kDLROCM>(device)
- .verify(dst_indices).verify(src_indices);
+ .verify(dst_indices);
+ const void* src_rows = nullptr;
+ if (src_indices.has_value()) {
+ TensorMatcher({L}).with_dtype<int32_t, int64_t>(indices_dtype).with_device<kDLCUDA, kDLROCM>(device)
+ .verify(src_indices.value());
+ src_rows = src_indices.value().data_ptr();
+ }
const int64_t* valid_length = nullptr;
if (num_indices.has_value()) {
@@ -582,8 +603,9 @@ struct MultiIndexCopyKernel {
static_cast<const int64_t*>(src_ptrs.data_ptr()),
static_cast<const int64_t*>(feat_bytes.data_ptr()),
dst_indices.data_ptr(),
- src_indices.data_ptr(),
+ src_rows,
valid_length,
+ reinterpret_cast<const volatile int64_t*>(wait_flag),
static_cast<int64_t>(L.unwrap()),
num_banks,
};
diff --git a/python/freetoken/kernel/csrc/pinned_tensor.cpp b/python/freetoken/kernel/csrc/pinned_tensor.cpp
index 9355f57a..1cdee5a6 100644
--- a/python/freetoken/kernel/csrc/pinned_tensor.cpp
+++ b/python/freetoken/kernel/csrc/pinned_tensor.cpp
@@ -60,6 +60,10 @@ torch::Tensor alloc_pinned_tensor(std::vector<int64_t> sizes,
void *data_ptr = nullptr;
const cudaError_t alloc_err = cudaHostAlloc(
&data_ptr, alloc_nbytes, cudaHostAllocPortable | cudaHostAllocMapped);
+ if (alloc_err != cudaSuccess) {
+ // callers may catch the refusal and go on: leave no error for the next launch
+ (void)cudaGetLastError();
+ }
TORCH_CHECK(alloc_err == cudaSuccess,
"cudaHostAlloc failed: ", cudaGetErrorString(alloc_err));
@@ -98,10 +102,20 @@ void host_register(int64_t addr, int64_t nbytes) {
const cudaError_t err =
cudaHostRegister(reinterpret_cast<void *>(addr), static_cast<size_t>(nbytes),
cudaHostRegisterPortable | cudaHostRegisterMapped);
+ if (err != cudaSuccess) {
+ // a refused pin is expected (the pin probe), so clear it before a later launch check reports it
+ (void)cudaGetLastError();
+ }
TORCH_CHECK(err == cudaSuccess,
"cudaHostRegister failed: ", cudaGetErrorString(err));
}
+void host_unregister(int64_t addr) {
+ const cudaError_t err = cudaHostUnregister(reinterpret_cast<void *>(addr));
+ TORCH_CHECK(err == cudaSuccess,
+ "cudaHostUnregister failed: ", cudaGetErrorString(err));
+}
+
int64_t driver_cuda_version() {
int version = 0; // stays 0 when no driver is installed
const cudaError_t err = cudaDriverGetVersion(&version);
@@ -123,6 +137,8 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
"Device-visible alias of a pinned+mapped host address");
m.def("host_register", &host_register,
"cudaHostRegister an existing host range as portable+mapped");
+ m.def("host_unregister", &host_unregister,
+ "cudaHostUnregister a range host_register pinned");
m.def("driver_cuda_version", &driver_cuda_version,
"Max CUDA version the installed NVIDIA driver supports (0 if none)");
}
diff --git a/python/freetoken/kernel/fast_index_copy.py b/python/freetoken/kernel/fast_index_copy.py
index b2030620..38be995f 100644
--- a/python/freetoken/kernel/fast_index_copy.py
+++ b/python/freetoken/kernel/fast_index_copy.py
@@ -172,11 +172,12 @@ def fast_index_copy_multi_jit(
src_ptrs: torch.Tensor,
feat_bytes: torch.Tensor,
dst_indices: torch.Tensor,
- src_indices: torch.Tensor,
+ src_indices: torch.Tensor | None,
num_indices: torch.Tensor | None = None,
*,
num_threads: int = 1024,
blocks_per_bank: int = 8,
+ wait_flag: int = 0,
) -> None:
"""Fused multi-bank index copy: copy the same rows for every bank in ONE launch.
@@ -193,13 +194,16 @@ def fast_index_copy_multi_jit(
once by the caller (the per-bank slot-cache base addr, host-source base addr, and
per-row byte size). Every bank's per-row byte size must be a multiple of 16, and the
base addresses 16-byte aligned (true for contiguous torch allocations of these banks).
+
+ ``src_indices`` None copies source row i to ``dst_indices[i]``. ``wait_flag``, a GPU-visible
+ int64 address, makes every block first wait until it holds >= 1 (rows a CPU thread writes).
"""
if _skip_fast_index_copy_enabled():
return
module = _jit_fast_index_copy_multi_module(
num_threads=num_threads, blocks_per_bank=blocks_per_bank
)
- module.launch(dst_ptrs, src_ptrs, feat_bytes, dst_indices, src_indices, num_indices)
+ module.launch(dst_ptrs, src_ptrs, feat_bytes, dst_indices, src_indices, num_indices, wait_flag)
def update_copy_flag_jit(sync_flag: torch.Tensor, delta: int) -> None:
diff --git a/python/freetoken/kernel/pinned.py b/python/freetoken/kernel/pinned.py
index a21e78ac..897e5d6f 100644
--- a/python/freetoken/kernel/pinned.py
+++ b/python/freetoken/kernel/pinned.py
@@ -50,6 +50,11 @@ def host_register(addr: int, nbytes: int) -> None:
_load_pinned_extension().host_register(addr, nbytes)
+def host_unregister(addr: int) -> None:
+ """cudaHostUnregister a range :func:`host_register` pinned."""
+ _load_pinned_extension().host_unregister(addr)
+
+
@lru_cache(maxsize=1)
def _host_ptr_identity() -> bool:
# cached per process: FreeToken pins one CUDA device per process (set at engine launch)
diff --git a/python/freetoken/moe/bench_profile.py b/python/freetoken/moe/bench_profile.py
index c939a947..319655cd 100644
--- a/python/freetoken/moe/bench_profile.py
+++ b/python/freetoken/moe/bench_profile.py
@@ -128,6 +128,19 @@ def fetch_fraction(row: dict | None) -> float | None:
return None
+def locked_fetch_fraction(row: dict | None) -> float | None:
+ """The fetch split of a locked layer of this row's kind: p / (2p + c) from the same pair as ``fetch_fraction``.
+
+ A locked fetch is a CPU memcpy into staging, then the PCIe gather; taking the memcpy rate as the CPU MoE rate c, the two steps in series run at p*c / (p + c), which against the CPU compute gives p / (2p + c)."""
+ if not isinstance(row, dict):
+ return None
+ for cpu_key, pcie_key in (("cpu_moe_overlap_gbs", "pcie_gather_overlap_gbs"), ("cpu_moe_gbs", "pcie_gather_gbs")):
+ cpu, pcie = row.get(cpu_key), row.get(pcie_key)
+ if cpu and pcie:
+ return pcie / (2 * pcie + cpu)
+ return None
+
+
def load_layer_kinds(gpu_name: str | None = None, path: str | None = None, gpu_uuid: str | None = None) -> dict | None:
"""The rows of the usable profile for this GPU (key -> row), or None without one."""
prof = _usable_profile(gpu_name, path, gpu_uuid)
diff --git a/python/freetoken/moe/expert_banks.py b/python/freetoken/moe/expert_banks.py
index b79d2860..dcb79797 100644
--- a/python/freetoken/moe/expert_banks.py
+++ b/python/freetoken/moe/expert_banks.py
@@ -54,6 +54,8 @@ class ExpertBanks:
row_tail_bytes: dict[str, list[int]] | None = None
# per-layer quant_format of experts in several formats (``quant_format``/``kind``/``kernel`` then name the most common one); None = every layer is ``quant_format``
layer_formats: list[str] | None = None
+ # the residency plan holding DEFERRED layers' banks, for host_banks.probe_pin; None when nothing was deferred
+ deferred_plan: object | None = None
def _dummy_fill(role: str, tensor: torch.Tensor) -> None:
@@ -439,6 +441,45 @@ def load_expert_banks(
return _echo_residency(banks, layer_residency, residency_plan)
+def pin_probe_enabled(config, mixed_formats: bool) -> bool:
+ """Whether ``--moe-cpu-layers auto`` finds the pin limit by pinning until refused.
+
+ Only where pinning is known to be capped; FREETOKEN_PIN_BUDGET_GB fixes the limit instead (then a failed pin is an error)."""
+ from freetoken.checkpoint.ftw import is_ftw_checkpoint
+ from freetoken.moe.host_banks import pin_capped
+
+ return (
+ (config.moe_cpu_layers or "").strip() == "auto"
+ and pin_capped()
+ and not os.environ.get("FREETOKEN_PIN_BUDGET_GB")
+ and config.moe_strategy in ("offload", "hybrid")
+ and not mixed_formats
+ and not config.use_dummy_weight
+ and not (config.model_path and is_ftw_checkpoint(config.model_path)) # FTW settles its banks itself
+ )
+
+
+def probe_banks(banks: ExpertBanks, num_layers: int) -> tuple[ExpertBanks, frozenset[int]]:
+ """Settle DEFERRED banks with host_banks.probe_pin; returns the banks and the OS-locked layers."""
+ import dataclasses
+ import time
+
+ from freetoken.moe.host_banks import HostResidency, probe_pin
+
+ plan = banks.deferred_plan
+ if plan is None:
+ return banks, frozenset() # the loader settled the banks itself (pinned)
+ t0 = time.perf_counter()
+ labels = probe_pin(plan, num_layers)
+ locked = frozenset(i for i, r in enumerate(labels) if r != HostResidency.PINNED.value)
+ pinned_bytes = sum(b.nbytes for l, bs in plan.deferred.items() if l not in locked for b in bs)
+ logger.info_rank0(
+ f"--moe-cpu-layers auto (pin probe): pinned {num_layers - len(locked)} MoE layers ({pinned_bytes / 2**30:.2f} GiB) "
+ f"in {time.perf_counter() - t0:.1f}s, locked {len(locked)} {sorted(locked)}"
+ )
+ return dataclasses.replace(banks, layer_residency=labels, deferred_plan=None), locked
+
+
def _echo_residency(banks: ExpertBanks, requested, plan) -> ExpertBanks:
"""Stamp an honored residency request onto the ExpertBanks; keep None (and warn) when no settle point consulted the plan."""
if requested is None or banks.layer_residency is not None:
@@ -446,6 +487,8 @@ def _echo_residency(banks: ExpertBanks, requested, plan) -> ExpertBanks:
if plan is not None and plan.applied:
import dataclasses
+ if plan.deferred:
+ return dataclasses.replace(banks, layer_residency=list(requested), deferred_plan=plan)
labels = [plan.actual.get(i, r) for i, r in enumerate(requested)]
downgraded = [i for i, r in enumerate(requested) if labels[i] != r]
if downgraded:
diff --git a/python/freetoken/moe/host_banks.py b/python/freetoken/moe/host_banks.py
index 95562a88..3cdc8d77 100644
--- a/python/freetoken/moe/host_banks.py
+++ b/python/freetoken/moe/host_banks.py
@@ -23,6 +23,7 @@ import math
import mmap
import os
import queue
+import sys
import threading
from concurrent.futures import ThreadPoolExecutor
from enum import Enum
@@ -50,6 +51,34 @@ class HostResidency(str, Enum):
PINNED = "pinned"
LOCKED = "locked"
PAGEABLE = "pageable"
+ # filled but not settled: probe_pin settles the layer after the load (--moe-cpu-layers auto finds the pin limit)
+ DEFERRED = "deferred"
+
+
+# FREETOKEN_SIMULATE_PIN_CAP_GB fails pins past that many GiB, like a driver pin quota (WSL/WDDM), to exercise the pin probe on hosts without one
+_sim_pinned = 0
+
+
+def _sim_pin_cap() -> int | None:
+ value = os.environ.get("FREETOKEN_SIMULATE_PIN_CAP_GB")
+ return int(float(value) * 2**30) if value else None
+
+
+def charge_pin(nbytes: int) -> None:
+ """Count ``nbytes`` against the simulated pin cap; raise PinFailed past it (no-op without the cap)."""
+ global _sim_pinned
+ cap = _sim_pin_cap()
+ if cap is None:
+ return
+ if _sim_pinned + nbytes > cap:
+ raise PinFailed(f"simulated pin cap {cap / 2**30:.1f} GiB reached")
+ _sim_pinned += nbytes
+
+
+def refund_pin(nbytes: int) -> None:
+ global _sim_pinned
+ if _sim_pin_cap() is not None:
+ _sim_pinned -= nbytes
_DEFAULT_CHUNK = 8 << 20
@@ -137,12 +166,24 @@ class HostBank:
return
from freetoken.kernel.pinned import host_register
+ charge_pin(len(self._buf))
try:
host_register(self.addr, len(self._buf))
except RuntimeError as exc:
+ refund_pin(len(self._buf))
raise PinFailed(f"cudaHostRegister failed for {len(self._buf) / 2**30:.1f} GiB") from exc
self._pinned = True
+ def unpin(self) -> None:
+ """Undo ``pin()`` on an mmap bank; its pages stay resident (born-pinned banks cannot unpin)."""
+ if not self._pinned or not isinstance(self._buf, mmap.mmap):
+ return
+ from freetoken.kernel.pinned import host_unregister
+
+ host_unregister(self.addr)
+ refund_pin(len(self._buf))
+ self._pinned = False
+
def release(self) -> None:
"""Drop the resident pages; the address space stays valid, the contents become undefined.
@@ -152,7 +193,7 @@ class HostBank:
self._buf.madvise(mmap.MADV_DONTNEED)
def lock(self) -> None:
- """mlock the (now-filled) buffer: resident without CUDA pin quota, but no device address -- only the CPU executor can serve a locked layer.
+ """mlock the (now-filled) buffer: resident without CUDA pin quota, but with no device address, so the GPU cannot gather its rows.
Lock after fill, or the lazy mmap faults+zero-fills every page. A failed lock (RLIMIT_MEMLOCK) warns once and leaves the bank PAGEABLE, which every consumer treats the same."""
if self._locked or self._pinned: # cudaHostRegister already page-locks
@@ -209,13 +250,15 @@ class _ResidencyPlan:
Installed by ``load_expert_banks`` around the provider dispatch so every loader honors --moe-cpu-layers without a new parameter in each signature. ``applied`` flips once a settle point consults the plan."""
- __slots__ = ("labels", "applied", "has_unpinned", "actual")
+ __slots__ = ("labels", "applied", "has_unpinned", "actual", "deferred")
def __init__(self, labels: list[str]):
self.labels = list(labels)
self.applied = False
self.has_unpinned = any(r != HostResidency.PINNED.value for r in labels)
self.actual: dict[int, str] = {}
+ # DEFERRED layers' filled banks, for probe_pin
+ self.deferred: dict[int, list[HostBank]] = {}
def residency_for(self, layer_id: int) -> str:
self.applied = True
@@ -264,6 +307,9 @@ def pin_banks(banks: dict[str, HostBank | list[HostBank] | dict[int, HostBank]])
HostResidency.PINNED.value if plan is None
else plan.residency_for(layer_id)
)
+ if residency == HostResidency.DEFERRED.value:
+ plan.deferred.setdefault(layer_id, []).append(layer_bank)
+ continue
_settle(layer_bank, residency)
if plan is not None and residency == HostResidency.LOCKED.value:
plan.record(layer_id, layer_bank.residency.value)
@@ -271,6 +317,61 @@ def pin_banks(banks: dict[str, HostBank | list[HostBank] | dict[int, HostBank]])
bank.pin()
+def _wddm() -> bool:
+ return sys.platform == "win32" or (hasattr(os, "uname") and "microsoft" in os.uname().release.lower())
+
+
+def pin_capped() -> bool:
+ """Whether CUDA pinning is known to be capped: WDDM (WSL and native Windows), or FREETOKEN_SIMULATE_PIN_CAP_GB."""
+ return _wddm() or _sim_pin_cap() is not None
+
+
+def _probe_margin() -> int:
+ """FREETOKEN_PIN_PROBE_MARGIN_GB, else 3 GiB under WDDM (WSL and native Windows)."""
+ return int(float(os.environ.get("FREETOKEN_PIN_PROBE_MARGIN_GB", "3" if _wddm() else "0")) * 2**30)
+
+
+def probe_pin(plan: _ResidencyPlan, num_layers: int, margin_bytes: int | None = None) -> list[str]:
+ """Settle the DEFERRED layers: pin middle layers first until a pin fails, OS-lock the rest; returns each layer's residency.
+
+ After a refusal, the last pinned layers give back at least ``margin_bytes`` (default ``_probe_margin()``)."""
+ if margin_bytes is None:
+ margin_bytes = _probe_margin()
+ # per-layer decode miss rates are U-shaped, so the head and tail layers are the ones left unpinned
+ order = sorted(plan.deferred, key=lambda l: (abs(l - (num_layers - 1) / 2), l))
+ pinned: list[int] = []
+ refused = None
+ for layer in order:
+ banks = plan.deferred[layer]
+ try:
+ for bank in banks:
+ bank.pin()
+ except PinFailed as exc:
+ for bank in banks:
+ bank.unpin()
+ refused = exc
+ break
+ pinned.append(layer)
+ if refused is not None:
+ freed = 0
+ while pinned and freed < margin_bytes:
+ layer = pinned.pop()
+ for bank in plan.deferred[layer]:
+ freed += len(bank._buf) if bank._pinned else 0
+ bank.unpin()
+ labels = [HostResidency.PINNED.value] * num_layers
+ kept = set(pinned)
+ for layer, banks in plan.deferred.items():
+ if layer in kept:
+ continue
+ for bank in banks:
+ bank.lock()
+ labels[layer] = (
+ HostResidency.LOCKED.value if all(b.residency is HostResidency.LOCKED for b in banks) else HostResidency.PAGEABLE.value
+ )
+ return labels
+
+
class PinPipeline:
"""Settle (pin or lock) filled banks while other banks are still being read.
@@ -297,6 +398,9 @@ class PinPipeline:
if self._exc is not None:
continue # drain without settling after a failure
bank, residency, plan, layer_id = item
+ if residency == HostResidency.DEFERRED.value:
+ plan.deferred.setdefault(layer_id, []).append(bank)
+ continue
try:
_settle(bank, residency)
if plan is not None and residency == HostResidency.LOCKED.value:
diff --git a/python/freetoken/moe/locked_fetch.py b/python/freetoken/moe/locked_fetch.py
new file mode 100644
index 00000000..b2e4331e
--- /dev/null
+++ b/python/freetoken/moe/locked_fetch.py
@@ -0,0 +1,235 @@
+"""Locked-layer fetch for hybrid decode.
+
+A locked layer's host banks are OS-locked but not registered with CUDA (the pin quota ran out), so the GPU cannot
+gather their rows. Each decode step a publish kernel hands the fetch list chosen by the hybrid ensure to CPU threads
+(``_cpu_moe.LockedFetch``), which copy each fetched row of every bank into pinned staging, and fast_index_copy_multi
+copies staging row i into ``evict_slots[i]``. The misses it does not fetch go to the CPU executor as on any hybrid layer.
+"""
+
+from __future__ import annotations
+
+import collections
+import os
+
+import torch
+
+from freetoken.utils import init_logger
+
+logger = init_logger(__name__)
+
+
+def locked_fetch_wanted(config) -> bool:
+ """Whether locked layers fetch through pinned staging: hybrid, --moe-locked-fetch not 0, CUDA (the copy kernels are untested on ROCm)."""
+ return config.moe_strategy == "hybrid" and config.moe_locked_fetch != 0 and torch.version.hip is None
+
+
+def check_locked_fetch_config(config) -> None:
+ """Reject --moe-locked-max-fetch past the experts of a layer before the banks are read (only with --moe-cpu-layers)."""
+ num_experts = config.model_config.num_experts
+ if config.moe_cpu_layers and locked_fetch_wanted(config) and not 1 <= config.moe_locked_max_fetch <= num_experts:
+ raise ValueError(f"--moe-locked-max-fetch {config.moe_locked_max_fetch} must be in [1, {num_experts}] (the experts of a layer)")
+
+
+def staging_reserve(config, bank_bytes) -> int:
+ """Pin budget the staging takes when --moe-cpu-layers is settled before the load: ``max_fetch`` rows of twice the mean expert.
+
+ ``bank_bytes()`` estimates the banks; it runs only when the staging is wanted."""
+ if not (config.moe_cpu_layers and locked_fetch_wanted(config)) or not (total := bank_bytes()):
+ return 0
+ mc = config.model_config
+ return 2 * (total // (mc.num_moe_layers * mc.num_experts)) * config.moe_locked_max_fetch
+
+
+def settle_banks(config, banks, *, probe: bool, has_locked: bool):
+ """Pin the staging when locked layers may fetch, then with ``probe`` settle the DEFERRED banks (expert_banks.probe_banks).
+
+ Pinning the staging first keeps the banks from taking its share of the pin budget. Returns the banks, the locked layers
+ the probe found (None without ``probe``) and the staging (None when unused); a staging that cannot be pinned raises."""
+ from freetoken.moe.expert_banks import probe_banks
+
+ staging = None
+ if locked_fetch_wanted(config) and (probe or has_locked):
+ staging = pin_staging(banks.sources, config.moe_locked_max_fetch)
+ locked = None
+ if probe:
+ banks, locked = probe_banks(banks, config.model_config.num_moe_layers)
+ if staging is not None and not locked:
+ refund_staging(staging) # every bank got pinned
+ staging = None
+ return banks, locked, staging
+
+
+def locked_pool_tags(layer_tags: list | None, locked, num_layers: int) -> list:
+ """Bank layer tags with the locked layers marked, so they group into slot pools of their own."""
+ return [(t, "locked") if i in locked else t for i, t in enumerate(layer_tags or [None] * num_layers)]
+
+
+def attach_locked_fetch(cache, config, staging: dict[str, torch.Tensor], layer_rows) -> None:
+ """Set ``cache.locked_fetch`` for ``cache.locked_hybrid``; needs the cache's CPU executor. ``layer_rows`` as in fetch_plan."""
+ locked = sorted(cache.locked_hybrid)
+ plan = fetch_plan(config.moe_locked_fetch, locked, layer_rows)
+ cache.locked_fetch = LockedLayerFetch(cache, locked, plan, config.moe_locked_max_fetch, cache.cpu_executor, staging=staging)
+
+
+def staging_bytes(sources: dict[str, list[torch.Tensor]], max_fetch: int) -> dict[str, int]:
+ """Per canonical bank role: ``max_fetch`` rows of the bank's widest row over every layer."""
+ from freetoken.moe.legacy_format import canonical_role
+
+ out = {}
+ for name, per_layer in sources.items():
+ rows = [t[0].numel() * t.element_size() for t in per_layer if t.numel()]
+ out[canonical_role(name)] = max([16] + [r * max_fetch for r in rows])
+ return out
+
+
+def alloc_staging(sizes: dict[str, int]) -> dict[str, torch.Tensor]:
+ """Pinned staging per bank role."""
+ from freetoken.kernel.pinned import alloc_pinned_tensor
+ from freetoken.moe.host_banks import charge_pin, refund_pin
+
+ charge_pin(sum(sizes.values()))
+ try:
+ return {role: alloc_pinned_tensor(n, dtype=torch.uint8) for role, n in sizes.items()}
+ except BaseException:
+ refund_pin(sum(sizes.values()))
+ raise
+
+
+def pin_staging(sources: dict[str, list[torch.Tensor]], max_fetch: int) -> dict[str, torch.Tensor]:
+ """The staging for ``sources``; raises when it cannot be pinned."""
+ from freetoken.moe.host_banks import PinFailed
+
+ sizes = staging_bytes(sources, max_fetch)
+ try:
+ return alloc_staging(sizes)
+ except (PinFailed, RuntimeError) as exc:
+ raise RuntimeError(
+ f"could not pin the {sum(sizes.values()) / 2**20:.1f} MiB locked-layer staging ({exc}); free pinned host memory, "
+ "lower --moe-locked-max-fetch, or pass --moe-locked-fetch 0 to keep locked layers CPU-only"
+ ) from exc
+
+
+def refund_staging(staging: dict[str, torch.Tensor]) -> None:
+ """Give the staging's bytes back to the simulated pin cap; the memory itself goes with its last reference."""
+ from freetoken.moe.host_banks import refund_pin
+
+ refund_pin(sum(t.numel() for t in staging.values()))
+
+
+def fetch_plan(share: float, layers: list[int], layer_rows) -> dict[int, tuple[int, float]]:
+ """(max_fetch, fraction) per locked layer for the hybrid ensure.
+
+ ``share`` > 0 fetches that share of every step's misses; otherwise each layer takes p / (2p + c) from its `ft bench bw`
+ row, or a fixed cap of 1 without one. ``layer_rows()`` returns (row key, label) per MoE layer and the profile's rows, or None.
+ """
+ if share > 0:
+ logger.info_rank0(f"--moe-locked-fetch {share:g}: locked layers fetch this share of each step's misses")
+ return {layer: (0, share) for layer in layers}
+ from freetoken.moe.bench_profile import locked_fetch_fraction
+
+ rows = layer_rows()
+ keys, labels, rows = rows if rows is not None else ([None] * (max(layers) + 1), [""] * (max(layers) + 1), {})
+ fractions = {layer: locked_fetch_fraction(rows.get(keys[layer])) for layer in layers}
+ groups = collections.Counter((labels[layer], f) for layer, f in fractions.items())
+ if fetched := [f"{f:.1%} on {n} {label} layers" for (label, f), n in groups.items() if f is not None]:
+ logger.info_rank0(f"--moe-locked-fetch auto: locked layers fetch this share of each step's misses (benched p / (2p + c)): {'; '.join(fetched)}")
+ if capped := [layer for layer, f in fractions.items() if f is None]:
+ logger.warning_rank0(
+ f"--moe-locked-fetch auto: no `ft bench bw` row for locked layers {capped}; they fetch at most 1 expert per step. "
+ "`ft bench bw --model <checkpoint>` benches each layer kind"
+ )
+ return {layer: (0, f) if f is not None else (1, 0.0) for layer, f in fractions.items()}
+
+
+def _smt_siblings(cpus: list[int]) -> list[int]:
+ out = []
+ for cpu in cpus:
+ try:
+ with open(f"/sys/devices/system/cpu/cpu{cpu}/topology/thread_siblings_list") as f:
+ text = f.read().strip()
+ except OSError:
+ continue
+ for part in text.split(","):
+ lo, _, hi = part.partition("-")
+ out.extend(i for i in range(int(lo), int(hi or lo) + 1) if i != cpu)
+ return out
+
+
+def _copy_threads() -> int:
+ """FREETOKEN_LOCKED_FETCH_THREADS, else max(2, min(4, logical CPUs / 4)): a few threads already saturate memcpy."""
+ if env := os.environ.get("FREETOKEN_LOCKED_FETCH_THREADS"):
+ return max(1, int(env))
+ logical = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else (os.cpu_count() or 1)
+ return max(2, min(4, logical // 4))
+
+
+def _copy_cpus(executor, count: int) -> list[int]:
+ """CPUs for the copy threads: SMT siblings of the executor's workers, else physical cores the executor leaves free."""
+ from freetoken.moe.cpu_executor import physical_core_cpus
+
+ busy = set(executor.core_ids) | {getattr(executor, "_coord_core", -1)}
+ cpus = [c for c in dict.fromkeys(_smt_siblings(executor.core_ids)) if c not in busy]
+ cpus = cpus or [c for c in physical_core_cpus() if c not in busy]
+ return cpus[:count]
+
+
+class LockedLayerFetch:
+ """Staging and copy threads for the locked layers of one ``OffloadMoeCache``."""
+
+ def __init__(self, cache, layers: list[int], plan: dict[int, tuple[int, float]], max_fetch: int, executor, staging: dict[str, torch.Tensor] | None = None) -> None:
+ from freetoken.kernel import _cpu_moe
+ from freetoken.kernel.pinned import device_ptr
+ from freetoken.moe.legacy_format import canonical_role
+
+ if not cache._copy_fused_ok:
+ raise ValueError("locked layers fetch through the fused multi-bank copy, which this cache could not plan (bank alignment or FREETOKEN_FUSED_COPY=0)")
+ # (max_fetch, fraction) of each locked layer for the hybrid ensure; max_fetch caps any of them
+ self.plan = plan
+ self.max_fetch = max_fetch
+ roles = [canonical_role(name) for name in cache.bank_schema]
+ nbanks = len(cache.banks)
+ src = [[0] * nbanks for _ in range(cache.num_layers)]
+ row_bytes = [[0] * nbanks for _ in range(cache.num_layers)]
+ need = [16] * nbanks
+ for b, (per_layer, _) in enumerate(cache.banks):
+ for layer in layers:
+ rb = int(cache._row_bytes[cache.layer_layout[layer]][b])
+ if per_layer[layer].numel() == 0 or rb == 0:
+ continue
+ src[layer][b] = per_layer[layer].data_ptr()
+ row_bytes[layer][b] = rb
+ need[b] = max(need[b], rb * max_fetch)
+ if staging is None:
+ staging = alloc_staging(dict(zip(roles, need)))
+ for b, role in enumerate(roles):
+ assert staging[role].numel() >= need[b], (role, staging[role].numel(), need[b])
+ self.staging = [staging[role] for role in roles]
+ self.staging_ptrs = torch.tensor([device_ptr(t) for t in self.staging], dtype=torch.int64, device=cache.device)
+ cpus = _copy_cpus(executor, _copy_threads())
+ # the executor's doorbell decision (FREETOKEN_CPU_MOE_FLAG_SYNC and the memop probe); without memops the gather spins on the GPU
+ memops = bool(getattr(executor, "_flag_sync", False))
+ self._ext = _cpu_moe.LockedFetch(src, row_bytes, [t.data_ptr() for t in self.staging], max_fetch, cpus, memops)
+ self._wait_flag = 0 if memops else self._ext.done_word()
+ logger.info_rank0(
+ f"locked layers {layers}: GPU slots + fetch up to {max_fetch} misses per step through "
+ f"{sum(t.numel() for t in self.staging) / 2**20:.1f} MiB of pinned staging, {max(1, len(cpus))} copy thread(s) on cpus {cpus or 'any'}"
+ + ("" if memops else ", the GPU spin-waits on them (no stream memops)")
+ )
+
+ def copy(self, cache, layer_id: int) -> None:
+ """Stage and gather the rows the last hybrid ensure scheduled for ``layer_id``; graph capturable."""
+ from freetoken.kernel.fast_index_copy import fast_index_copy_multi_jit
+
+ # the gather reads num_indices rows of the staging, which has max_fetch
+ assert cache._pending_fetch_cap == self.max_fetch, (cache._pending_fetch_cap, self.max_fetch)
+ self._ext.publish(torch.cuda.current_stream().cuda_stream, layer_id, cache.num_indices.data_ptr(), cache.src_indices.data_ptr())
+ fast_index_copy_multi_jit(
+ cache._copy_dst_ptrs_pool[cache.pool_of_layer[layer_id]],
+ self.staging_ptrs,
+ cache._copy_feat_bytes[cache.layer_layout[layer_id]],
+ cache.evict_slots,
+ None,
+ cache.num_indices,
+ blocks_per_bank=cache._copy_blocks_per_bank,
+ wait_flag=self._wait_flag,
+ )
diff --git a/python/freetoken/moe/offload_cache.py b/python/freetoken/moe/offload_cache.py
index 0d702d8a..f555be96 100644
--- a/python/freetoken/moe/offload_cache.py
+++ b/python/freetoken/moe/offload_cache.py
@@ -194,6 +194,9 @@ class OffloadMoeCache:
# offload/PCIe path. Set by the engine after construction (empty = all-GPU,
# all layers = the plain --moe-strategy cpu case).
self.cpu_layer_ids: frozenset = frozenset()
+ # hybrid: locked (unpinned) layers that keep slots of their own pools and fetch part of their misses through locked_fetch's pinned staging
+ self.locked_hybrid: frozenset = frozenset()
+ self.locked_fetch = None # moe.locked_fetch.LockedLayerFetch, set by the engine with locked_hybrid
# Bank row layouts, set by set_bank_sources: layouts[g][k] is (dtype, row shape) of bank k (layouts of two quant tags may be equal), layer_layout[l] is layer l's g.
# With one (dtype, row shape) per bank a bank cache is typed [cache_size, *row]; with several, or with pads, it is a byte segment each pool views with its own layout.
self.layouts: list[tuple[tuple[torch.dtype, tuple[int, ...]], ...]] = []
@@ -256,7 +259,7 @@ class OffloadMoeCache:
self.bank_schema = _BANK_SCHEMAS[self.quant_format]
self.bank_sources: dict[str, list[torch.Tensor]] = {}
self.bank_caches: dict[str, torch.Tensor] = {}
- # per-layer host residency: the GPU movement paths require "pinned"; LOCKED/PAGEABLE layers decode on the CPU executor and prefill via copy_missing's pageable branch
+ # per-layer host residency: the GPU movement paths require "pinned"; LOCKED/PAGEABLE layers decode on the CPU executor (locked_hybrid ones also from their slots) and prefill via copy_missing's pageable branch
# _unpinned_layers is the derived id set the hot paths test against
self.layer_residency: list[str] = []
self._unpinned_layers: frozenset = frozenset()
@@ -314,6 +317,7 @@ class OffloadMoeCache:
# _pending_whole_layer records WHICH staged it: the pageable branch is only sound after materialize_layer
self._pending_src_layer: int | None = None
self._pending_whole_layer = False
+ self._pending_fetch_cap: int | None = None # the last hybrid ensure's cap on its fetch count
# The two prefill staging buffers are the first 2 * _stage_bytes[k] bytes of every bank, on top of the first pools.
# _stage_views[g][b] are buffer b's per-bank [num_experts, ...] views for layout g; _inval[b] are the global slots it overwrites.
self._stage_bytes: list[int] = []
@@ -468,7 +472,7 @@ class OffloadMoeCache:
-- the cache machinery is layout-agnostic and just moves rows.
``layer_residency`` labels each layer with a ``HostResidency`` value (default: all pinned).
- Non-pinned (LOCKED/PAGEABLE) layers have no device address: they must already be routed to the CPU executor (``cpu_layer_ids``, set BEFORE this call), the copy plan skips their rows, and their only movement is ``copy_missing``'s whole-layer pageable prefill branch -- which is why prefill overlap is incompatible with them.
+ Non-pinned (LOCKED/PAGEABLE) layers have no device address: they must be in ``cpu_layer_ids`` (set BEFORE this call) and the copy plan skips their rows. Prefill moves them with ``copy_missing``'s whole-layer pageable branch, so prefill overlap is incompatible with them; ``locked_hybrid`` layers also fetch decode misses through ``locked_fetch``.
``layer_tags`` names each layer's quant (see :func:`bank_layouts`); layers of different tags never share a pool.
@@ -784,11 +788,11 @@ class OffloadMoeCache:
def is_cpu_layer(self, layer_id: int) -> bool:
"""Whether ``layer_id`` decodes on the CPU executor (vs the GPU offload path)."""
- return layer_id in self.cpu_layer_ids
+ return layer_id in self.cpu_layer_ids and layer_id not in self.locked_hybrid
def is_unpinned_layer(self, layer_id: int) -> bool:
"""Whether ``layer_id``'s host banks have no device address (LOCKED/PAGEABLE): the GPU slot-gather paths cannot serve it.
- ``copy_missing`` takes the whole-layer pageable branch, which presumes materialize's position == expert id (never ``ensure_experts``'s LRU slot remap)."""
+ ``copy_missing`` takes the whole-layer pageable branch, which presumes materialize's position == expert id; a locked_hybrid layer's decode fetches go through ``locked_fetch`` instead."""
return layer_id in self._unpinned_layers
def alphas_for_slots(self, layer_id: int) -> tuple[torch.Tensor, torch.Tensor] | None:
@@ -1096,7 +1100,12 @@ class OffloadMoeCache:
self._pending_src_layer = layer_id
self._pending_whole_layer = False
fraction = self.hybrid_fetch_fraction[layer_id] if self.hybrid_fetch_fraction is not None else 0.0
- return ensure_experts_hybrid(self, layer_id, expert_ids, weights, self.hybrid_max_fetch, fraction, clamp)
+ max_fetch, cap = self.hybrid_max_fetch, None
+ if layer_id in self.locked_hybrid:
+ # a locked layer fetches through the staging, so at most its rows per step
+ (max_fetch, fraction), cap = self.locked_fetch.plan[layer_id], self.locked_fetch.max_fetch
+ self._pending_fetch_cap = cap
+ return ensure_experts_hybrid(self, layer_id, expert_ids, weights, max_fetch, fraction, clamp, cap)
def materialize_layer(self, layer_id: int) -> None:
from freetoken.moe.offload_kernels import materialize_layer
@@ -1156,6 +1165,29 @@ class OffloadMoeCache:
"prefill_rows": self.prefill_total_rows,
}
+ def decode_stats_line(self) -> str | None:
+ """Decode stats since the last call by layer group (pinned, locked with slots, CPU-only), as one log line; zeroes the counters."""
+ if self.decode_target == "hybrid":
+ rows = torch.stack([self.stat_steps_layer, self.stat_active_layer, self.stat_missing_layer, self.stat_fetched_layer], 1).tolist()
+ else:
+ # offload copies every miss
+ cols = self.lru_stats.t()
+ rows = torch.stack([cols[Stat.CALLS], cols[Stat.ACTIVE], cols[Stat.MISS], cols[Stat.MISS]], 1).tolist()
+ self.reset_stats()
+ groups = (("pinned", [l for l in range(self.num_layers) if l not in self.cpu_layer_ids]), ("locked", sorted(self.locked_hybrid)))
+ parts = []
+ for name, layers in groups:
+ steps, active, missing, fetched = (sum(rows[l][k] for l in layers) for k in range(4))
+ if steps and active:
+ parts.append(
+ f"{name} {_layers(len(layers))}: hit {1 - missing / active:.1%}, per layer-step "
+ f"miss {missing / steps:.2f} fetch {fetched / steps:.2f} cpu {(missing - fetched) / steps:.2f}"
+ )
+ cpu_only = self.cpu_layer_ids - self.locked_hybrid
+ if cpu_only:
+ parts.append(f"cpu-only {_layers(len(cpu_only))}")
+ return "MoE decode, " + "; ".join(parts) if parts else None
+
def decode_miss_stats_per_layer(self) -> dict:
"""Per-MoE-layer realized decode stats for one (reset_stats-delimited) window.
@@ -1224,13 +1256,16 @@ class OffloadMoeCache:
assert layer_id is not None, "no staged misses (ensure_experts/materialize_layer first)"
if layer_id in self._unpinned_layers:
if not self._pending_whole_layer:
+ if layer_id in self.locked_hybrid:
+ self.locked_fetch.copy(self, layer_id)
+ return
raise RuntimeError(
f"layer {layer_id} is unpinned: its only copy is the whole-layer "
f"pageable materialize (position == expert id); ensure_experts's "
f"LRU slot remap cannot be honored without a device alias"
)
# the only copy a non-pinned layer ever needs is the non-overlap prefill materialize, which schedules the whole layer into staging buffer 0 with position == expert id -- a plain synchronous pageable H2D copy
- # never CUDA-graph captured: prefill is not captured, and decode never reaches this branch (it routes to the CPU executor)
+ # never CUDA-graph captured: prefill is not captured, and decode of an unpinned layer routes to the CPU executor or, locked_hybrid, to the staged fetch above
for (per_layer, _), view in zip(self.banks, self.stage_views(layer_id, 0)):
view.copy_(per_layer[layer_id])
return
@@ -1269,6 +1304,10 @@ class OffloadMoeCache:
)
+def _layers(n: int) -> str:
+ return f"{n} layer" if n == 1 else f"{n} layers"
+
+
def iter_offload_moe_layers(model) -> Iterator:
from freetoken.layers import OffloadMoELayer, iter_moe_layers
diff --git a/python/freetoken/moe/offload_kernels.py b/python/freetoken/moe/offload_kernels.py
index 8c4ad8f3..5b28c1be 100644
--- a/python/freetoken/moe/offload_kernels.py
+++ b/python/freetoken/moe/offload_kernels.py
@@ -58,6 +58,7 @@ def ensure_experts_hybrid(
max_fetch: int,
fetch_fraction: float = 0.0,
clamp: bool = False,
+ fetch_cap: int | None = None,
) -> HybridRoutes:
"""Capped-fetch variant of ``ensure_experts`` (hybrid backend).
@@ -71,13 +72,15 @@ def ensure_experts_hybrid(
pre-cap miss count (stats).
The same launch splits the routes: ``expert_ids`` becomes the GPU slots in place, and
- ``clamp`` gives CPU routes slot 0 instead of -1 for GEMM kernels that read every slot."""
+ ``clamp`` gives CPU routes slot 0 instead of -1 for GEMM kernels that read every slot.
+ ``fetch_cap`` bounds the fetch count after ``max_fetch`` / ``fetch_fraction`` (a fixed-size staging buffer)."""
assert expert_ids.is_cuda, "the hybrid ensure kernel runs on the GPU"
# Q16 fixed point: the kernel caps in integers.
frac_q16 = min(1 << 16, max(0, round(fetch_fraction * (1 << 16))))
weights = weights.contiguous()
routes = HybridRoutes(expert_ids, torch.empty_like(weights), torch.empty_like(expert_ids))
- _ensure_experts_hybrid_gpu(cache, layer_id, expert_ids, weights, routes, clamp, max_fetch, frac_q16)
+ cap = cache.num_experts if fetch_cap is None else fetch_cap
+ _ensure_experts_hybrid_gpu(cache, layer_id, expert_ids, weights, routes, clamp, max_fetch, frac_q16, cap)
return routes
@@ -132,6 +135,7 @@ def _ensure_experts_hybrid_gpu(
clamp: bool,
max_fetch: int,
frac_q16: int,
+ fetch_cap: int,
) -> None:
id_of_slot, usage = cache.pool_state(layer_id)
num_slots = id_of_slot.numel()
@@ -158,6 +162,7 @@ def _ensure_experts_hybrid_gpu(
expert_ids.numel(),
int(max_fetch),
int(frac_q16),
+ int(fetch_cap),
cache.num_experts,
num_slots,
BLOCK_E=block_e,
@@ -268,7 +273,7 @@ def _materialize_layer_kernel(
[email protected](do_not_specialize=["layer_id", "num_active", "max_fetch", "fetch_frac_q16", "cache_size", "id_of_slot_ptr", "usage_ptr"])
[email protected](do_not_specialize=["layer_id", "num_active", "max_fetch", "fetch_frac_q16", "fetch_cap", "cache_size", "id_of_slot_ptr", "usage_ptr"])
def _ensure_experts_hybrid_kernel(
expert_ids_ptr,
slot_for_id_ptr,
@@ -290,6 +295,7 @@ def _ensure_experts_hybrid_kernel(
num_active,
max_fetch,
fetch_frac_q16,
+ fetch_cap,
num_experts: tl.constexpr,
cache_size,
BLOCK_E: tl.constexpr,
@@ -342,7 +348,7 @@ def _ensure_experts_hybrid_kernel(
max_fetch = tl.where(cost_lo <= cost_hi, lo, lo + 1)
is_hit = is_active & (slot >= 0)
# a pool smaller than num_experts may hold fewer free slots than misses
- num_fetch = tl.minimum(tl.minimum(num_missing, max_fetch), cache_size - tl.sum(is_hit.to(tl.int32)))
+ num_fetch = tl.minimum(tl.minimum(num_missing, tl.minimum(max_fetch, fetch_cap)), cache_size - tl.sum(is_hit.to(tl.int32)))
tl.store(num_missing_full_ptr, num_missing.to(tl.int64))
tl.store(num_indices_ptr, num_fetch.to(tl.int64))
tl.store(usage_ptr + slot, step, mask=is_hit)
diff --git a/python/freetoken/scheduler/scheduler.py b/python/freetoken/scheduler/scheduler.py
index d057d3b9..220fa090 100644
--- a/python/freetoken/scheduler/scheduler.py
+++ b/python/freetoken/scheduler/scheduler.py
@@ -143,14 +143,21 @@ class Scheduler(SchedulerIOMixin):
self.config = config
self._model_is_mrope = config.model_config.model_is_mrope
self._warned_cut_image = False
+ moe_cache = getattr(self.engine, "moe_offload_cache", None)
self.status_reporter = SchedulerStatusReporter(
log=logger.info_rank0,
decode_log_interval=config.decode_log_interval,
+ moe_stats=self._moe_stats_line if config.moe_collect_stats and moe_cache is not None else None,
)
# Initialize the I/O mixin
super().__init__(config, self.engine.tp_cpu_group)
+ def _moe_stats_line(self) -> str | None:
+ # on the engine stream, so the read and the zeroing fall between decode replays
+ with torch.cuda.stream(self.engine.stream):
+ return self.engine.moe_offload_cache.decode_stats_line()
+
def run_when_idle(self) -> None:
"""Called when the scheduler is idle to perform background tasks."""
logger.info_rank0("Scheduler is idle, waiting for new reqs...")
diff --git a/python/freetoken/scheduler/status.py b/python/freetoken/scheduler/status.py
index ca706c50..76c20cd4 100644
--- a/python/freetoken/scheduler/status.py
+++ b/python/freetoken/scheduler/status.py
@@ -12,6 +12,8 @@ class SchedulerStatusReporter:
log: Callable[[str], None]
clock: Callable[[], float] = time.perf_counter
decode_log_interval: int = 40
+ # --moe-collect-stats: one more line per decode log line (the stats since the previous one), or None to skip
+ moe_stats: Callable[[], str | None] | None = None
_last_prefill_time: float = field(init=False)
_last_decode_time: float = field(init=False)
_decode_forward_count: int = field(default=0, init=False)
@@ -122,6 +124,8 @@ class SchedulerStatusReporter:
f"gen throughput (token/s): {gen_throughput:.2f}, "
f"#queue-req: {queue_reqs}"
)
+ if self.moe_stats is not None and (line := self.moe_stats()):
+ self.log(line)
def _usage_ratio(used: int, total: int) -> float:
diff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.py
index 56d52ee7..d09534a8 100644
--- a/python/freetoken/server/args.py
+++ b/python/freetoken/server/args.py
@@ -795,9 +795,11 @@ def parse_args(
"With --moe-strategy offload/hybrid: which MoE layers compute on the "
"CPU executor instead of the GPU offload/PCIe path (where CUDA pinning "
"is quota-capped, e.g. WSL, their banks are OS-locked instead of pinned). Explicit id list ('3,7,11'), a count ('8' = 8 "
- "layers evenly strided), a fraction ('0.5'), or 'auto'. 'auto' is for Windows/WSL "
- "only, where CUDA pinned memory is capped: it locks just enough head+tail layers "
- "for the banks over the pin budget. Any value, 'auto' included, commits to CPU "
+ "layers evenly strided), a fraction ('0.5'), or 'auto'. 'auto' is for hosts that cap CUDA "
+ "pinned memory (Windows/WSL): it pins the banks layer by layer from the middle out until CUDA "
+ "refuses one and OS-locks the rest. FREETOKEN_PIN_BUDGET_GB, FTW checkpoints and models the CPU executor "
+ "cannot serve lock head+tail layers from a size estimate instead. Under hybrid, locked layers still decode from GPU slots "
+ "(--moe-locked-fetch). Any value, 'auto' included, commits to CPU "
"decode before the model is built, so the expert format must have a CPU executor "
"path (bf16, fp8_block, nvfp4, mxfp4); do not pass it on Linux. Unset = every layer on the "
"GPU; a boot whose banks exceed a known pin budget stops and asks for this flag."
@@ -818,6 +820,29 @@ def parse_args(
),
)
+ parser.add_argument(
+ "--moe-locked-fetch",
+ type=float,
+ default=ServerArgs.moe_locked_fetch,
+ help=(
+ "For --moe-strategy hybrid with locked layers (banks over the pin quota, OS-locked instead of pinned): "
+ "the share of a locked layer's misses fetched per decode step into GPU slots of its own pools, through "
+ "pinned staging that CPU threads fill; the rest are computed on the CPU. -1 (default) = auto: per layer from "
+ "the `ft bench bw` row of its kind, p / (2p + c) of the overlapped PCIe and CPU MoE rates (at most 1 expert "
+ "per step without a row). 0 = locked layers decode on the CPU only."
+ ),
+ )
+
+ parser.add_argument(
+ "--moe-locked-max-fetch",
+ type=int,
+ default=ServerArgs.moe_locked_max_fetch,
+ help=(
+ "For --moe-locked-fetch: the most experts a locked layer fetches per decode step. Sizes the pinned "
+ "staging: this many rows of the widest expert, per bank."
+ ),
+ )
+
parser.add_argument(
"--disable-moe-prefill-overlap",
action="store_false",
@@ -845,6 +870,18 @@ def parse_args(
),
)
+ parser.add_argument(
+ "--moe-collect-stats",
+ action="store_true",
+ default=ServerArgs.moe_collect_stats,
+ help=(
+ "With the offload-family strategies: after each decode log line, log the decode MoE stats since the "
+ "previous one (hit rate, and per layer-step the misses, experts fetched to the GPU and experts computed "
+ "on the CPU) for pinned layers, locked layers with GPU slots and CPU-only layers. Adds counter updates "
+ "to every decode step."
+ ),
+ )
+
parser.add_argument(
"--moe-prefill-hit-d2d",
action="store_true",
diff --git a/setup.py b/setup.py
index 070b8fd5..a836038b 100644
--- a/setup.py
+++ b/setup.py
@@ -267,6 +267,8 @@ setup(
name="freetoken.kernel._cpu_moe",
sources=[
"python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp",
+ "python/freetoken/kernel/csrc/cpu_moe/locked_fetch.cpp",
+ "python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.cu",
f"python/freetoken/kernel/csrc/cpu_moe/gpu_sync_{'hip' if IS_ROCM else 'cuda'}.cpp",
"python/freetoken/kernel/csrc/cpu_moe/gpu_sync_io.cu",
*sorted(
diff --git a/tests/moe/test_fused_copy.py b/tests/moe/test_fused_copy.py
index 7a0c0864..54c42680 100644
--- a/tests/moe/test_fused_copy.py
+++ b/tests/moe/test_fused_copy.py
@@ -5,6 +5,8 @@ zero-copy case), across banks of differing per-row sizes.
from __future__ import annotations
+import time
+
import pytest
import torch
@@ -74,3 +76,34 @@ def test_fused_copy_matches_per_bank(num_indices):
for b, (r, (_, c)) in enumerate(zip(ref, cache.banks)):
assert torch.equal(r, c), f"bank {b} (feat={FEATS[b]}) fused != per-bank at num_indices={num_indices}"
+
+
+@CUDA
+def test_fused_copy_identity_rows_and_wait_flag():
+ # without src_indices row i copies source row i; with wait_flag the copy holds until the flag is set (the locked-layer staging)
+ from freetoken.kernel.fast_index_copy import fast_index_copy_multi_jit
+ from freetoken.kernel.pinned import alloc_pinned_tensor, device_ptr
+
+ dev = torch.device("cuda")
+ feats = [512, 256]
+ src = [alloc_pinned_tensor(4 * f, dtype=torch.uint8) for f in feats]
+ for t in src:
+ t.random_(0, 256)
+ dst = [torch.zeros(16 * f, dtype=torch.uint8, device=dev) for f in feats]
+ dst_ptrs = torch.tensor([t.data_ptr() for t in dst], dtype=torch.int64, device=dev)
+ src_ptrs = torch.tensor([device_ptr(t) for t in src], dtype=torch.int64, device=dev)
+ feat_bytes = torch.tensor(feats, dtype=torch.int64, device=dev)
+ slots = torch.tensor([5, 2, 9, 0], dtype=torch.int32, device=dev)
+ num = torch.tensor([3], dtype=torch.int64, device=dev)
+ flag = alloc_pinned_tensor(1, dtype=torch.int64)
+ flag.fill_(0)
+
+ fast_index_copy_multi_jit(dst_ptrs, src_ptrs, feat_bytes, slots, None, num, wait_flag=device_ptr(flag))
+ time.sleep(0.2)
+ assert not torch.cuda.current_stream().query(), "the copy ran before its flag was set"
+ flag.fill_(1)
+ torch.cuda.synchronize()
+ for d, s, f in zip(dst, src, feats):
+ for i, slot in enumerate([5, 2, 9]):
+ assert torch.equal(d[slot * f:(slot + 1) * f].cpu(), s[i * f:(i + 1) * f])
+ assert not d[0:f].any(), "num_indices bounds the rows"
diff --git a/tests/moe/test_hybrid_fetch.py b/tests/moe/test_hybrid_fetch.py
index 5f36ac7b..384a3524 100644
--- a/tests/moe/test_hybrid_fetch.py
+++ b/tests/moe/test_hybrid_fetch.py
@@ -9,7 +9,7 @@ import os
import pytest
-from freetoken.moe.bench_profile import PROFILE_VERSION, default_profile_path, fetch_fraction, layer_kind_key, load_layer_kinds
+from freetoken.moe.bench_profile import PROFILE_VERSION, default_profile_path, fetch_fraction, layer_kind_key, load_layer_kinds, locked_fetch_fraction
def test_layer_kind_rows_give_fetch_fractions(tmp_path):
@@ -32,6 +32,9 @@ def test_layer_kind_rows_give_fetch_fractions(tmp_path):
assert fetch_fraction(rows[bf16]) == pytest.approx(0.4)
# overlapped pair preferred: pcie_ov / (pcie_ov + cpu_ov)
assert fetch_fraction(rows[gguf]) == pytest.approx(0.25)
+ # a locked layer's fetch adds a memcpy at the CPU MoE rate before the gather: p / (2p + c), from the same pair
+ assert locked_fetch_fraction(rows[bf16]) == pytest.approx(40 / 180)
+ assert locked_fetch_fraction(rows[gguf]) == pytest.approx(30 / 150)
# another thread count is another kind
assert layer_kind_key("bf16", {}, 2048, 768, 8, 6) not in rows
# a profile from different hardware is ignored
diff --git a/tests/moe/test_offload.py b/tests/moe/test_offload.py
index 5d7d0ab8..8833cdf9 100644
--- a/tests/moe/test_offload.py
+++ b/tests/moe/test_offload.py
@@ -1030,6 +1030,101 @@ def test_locked_layer_copy_missing_rejects_ensure_experts_staging():
cache.copy_missing()
[email protected](not torch.cuda.is_available(), reason="needs CUDA")
[email protected]("memops", [True, False])
+def test_locked_hybrid_layer_fetches_through_staging_in_a_graph(monkeypatch, memops):
+ # a locked layer's hybrid decode fetches at most max_fetch misses per step into its slots through the pinned staging; the rest go to the CPU
+ # memops: the GPU waits on the doorbell with a stream memop, else the gather kernel spins on it
+ from types import SimpleNamespace
+
+ from freetoken.moe.locked_fetch import LockedLayerFetch
+
+ monkeypatch.setenv("FREETOKEN_LOCKED_FETCH_THREADS", "2")
+ torch.manual_seed(0)
+ cache, sources = _make_split_cache(num_layers=2, locked=(1,), device="cuda")
+ cache.decode_target = "hybrid"
+ cache.locked_hybrid = frozenset({1})
+ cache.locked_fetch = LockedLayerFetch(cache, [1], plan={1: (0, 1.0)}, max_fetch=2, executor=SimpleNamespace(core_ids=[], _coord_core=-1, _flag_sync=memops))
+
+ top_k = 3
+ ids = torch.zeros(top_k, dtype=torch.int32, device="cuda")
+ weights = torch.rand(top_k, device="cuda")
+ out = {}
+
+ def step():
+ slots, _, cpu_ids = cache.ensure_experts_hybrid(1, ids, weights)
+ cache.copy_missing()
+ out["slots"], out["cpu"] = slots, cpu_ids
+
+ cache.collect_stats = True
+ stream = torch.cuda.Stream()
+ stream.wait_stream(torch.cuda.current_stream())
+ with torch.cuda.stream(stream):
+ step()
+ torch.cuda.current_stream().wait_stream(stream)
+ cache.reset()
+ graph = torch.cuda.CUDAGraph()
+ with torch.cuda.graph(graph):
+ step()
+
+ def replay(graph, steps):
+ resident = set()
+ for _ in range(steps):
+ experts = torch.randperm(4)[:top_k].to(torch.int32)
+ ids.copy_(experts)
+ graph.replay()
+ torch.cuda.synchronize()
+ slots, cpu_ids = out["slots"].tolist(), out["cpu"].tolist()
+ fetched = [e for e, s in zip(experts.tolist(), slots) if s >= 0 and e not in resident]
+ assert len(fetched) <= 2
+ for e, s, c in zip(experts.tolist(), slots, cpu_ids):
+ assert (s >= 0) != (c >= 0)
+ if s >= 0:
+ for (src, _), view in zip(cache.banks, cache.pool_bank_views(1)):
+ assert torch.equal(view[s].cpu(), src[1][e])
+ resident = {e for e, s in enumerate(cache.slot_for_id[1].tolist()) if s >= 0}
+
+ replay(graph, 12)
+ # --moe-collect-stats: the locked layer reports as its own group, and each line clears the counters
+ line = cache.decode_stats_line()
+ assert "locked 1 layer: hit" in line and "pinned" not in line, line
+ assert cache.decode_stats_line() is None
+ # a runtime rebuild reallocates evict_slots and the slot caches; the fetch follows and a new capture still copies right
+ del graph
+ cache.rebuild(12)
+ graph = torch.cuda.CUDAGraph()
+ with torch.cuda.graph(graph):
+ step()
+ replay(graph, 6)
+
+
+def test_pin_probe_pins_middle_layers_until_refused(monkeypatch):
+ # --moe-cpu-layers auto: deferred banks pin middle-out until the (simulated) quota refuses one; the rest lock, and the margin gives back the last pinned layer
+ import freetoken.kernel.pinned as pinned
+ import freetoken.moe.host_banks as hb
+
+ monkeypatch.setattr(pinned, "host_register", lambda addr, nbytes: None)
+ monkeypatch.setattr(pinned, "host_unregister", lambda addr: None)
+ monkeypatch.setattr(hb, "_os_lock", lambda addr, nbytes: None)
+ monkeypatch.setattr(hb, "_os_lock_failed", False)
+ monkeypatch.setattr(hb, "_sim_pinned", 0)
+ monkeypatch.setenv("FREETOKEN_SIMULATE_PIN_CAP_GB", str(3 * hb._BLK / 2**30))
+ deferred = [hb.HostResidency.DEFERRED.value] * 6
+
+ def probe(margin):
+ banks = {"gate_up": [hb.HostBank((4,), torch.uint8) for _ in range(6)]}
+ with hb.requested_residency(deferred) as plan:
+ hb.pin_banks(banks)
+ labels = hb.probe_pin(plan, 6, margin_bytes=margin)
+ for bank in banks["gate_up"]:
+ bank.unpin()
+ return [i for i, r in enumerate(labels) if r == hb.HostResidency.LOCKED.value]
+
+ # order 2, 3, 1, 4, 0, 5: the quota takes three, layer 4 is refused
+ assert probe(0) == [0, 4, 5]
+ assert probe(hb._BLK) == [0, 1, 4, 5]
+
+
def test_requested_residency_routes_layer_settles(monkeypatch):
# the ambient plan installed by load_expert_banks must route each layer's banks by label at both slow-path settle points (PinPipeline layer sink, list-valued pin_banks) and record that it was consulted
# without a plan everything pins
diff --git a/tests/scheduler/test_scheduler_status.py b/tests/scheduler/test_scheduler_status.py
index dcaa5740..be609f65 100644
--- a/tests/scheduler/test_scheduler_status.py
+++ b/tests/scheduler/test_scheduler_status.py
@@ -159,3 +159,13 @@ def test_usage_ratio_guard():
assert _usage_ratio(0, 0) == 0.0
assert _usage_ratio(5, 0) == 0.0
assert _usage_ratio(5, 10) == 0.5
+
+
+def test_moe_stats_line_follows_each_decode_line():
+ logs: list[str] = []
+ calls = []
+ rep = SchedulerStatusReporter(log=logs.append, clock=lambda: 1.0, decode_log_interval=2, moe_stats=lambda: calls.append(1) or "MoE decode, x")
+ for _ in range(4):
+ rep.report_batch(_decode_batch(1), running_reqs=1, queue_reqs=0, kv_used_pages=1, kv_total_pages=10, page_size=1)
+ assert len(calls) == 2
+ assert [line.split(",")[0] for line in logs] == ["Decode batch", "MoE decode"] * 2