FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
docs: add a guide to speeding up prefill with offloaded experts (#785)
Xiaoze Fanb10c0c479bae
diff --git a/docs/cli.md b/docs/cli.mdindex 34a9ffaa..5a759374 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -46,7 +46,7 @@ parsers all resolve automatically from the checkpoint and the GPU.| `--max-running-requests` | 4 | Max concurrently running requests || `--max-output-tokens` | 32768 | Default output budget for requests that omit one || `--max-seq-len-override` | from checkpoint | Max sequence length |-| `--max-prefill-length` | 8192 | Chunked-prefill chunk size in tokens |+| `--max-prefill-length` | 8192 | Chunked-prefill chunk size in tokens; with offloaded experts see [performance/prefill.md](performance/prefill.md) || `--cuda-graph-max-bs`, `--graph` | = max running requests | Max batch size captured as CUDA graphs || `--decode-log-interval` | 40 | Scheduler status line every N decode steps |@@ -77,7 +77,7 @@ ft serve --model ... --gpu GPU-9e8d7c6b # the same card by UUID (a unique prefi### MoE offload-See [models.md](models.md#moe-strategies) for what each strategy does.+See [models.md](models.md#moe-strategies) for what each strategy does, and [performance/prefill.md](performance/prefill.md) for prefill speed.| Flag | Default | Meaning ||---|---|---|diff --git a/docs/performance/prefill.md b/docs/performance/prefill.mdnew file mode 100644index 00000000..044880db--- /dev/null+++ b/docs/performance/prefill.md@@ -0,0 +1,153 @@+# Speeding up prefill with offloaded experts++This guide applies to the `offload` and `hybrid` MoE strategies (see+[models.md](../models.md#moe-strategies)). With `fused` every expert is already on+the GPU and none of this applies.++Prefill speed has two sides: **copy** (moving experts over PCIe) and+**compute** (the forward pass). With overlap on, a chunk takes as long as the+slower of the two, so first find which one is the bottleneck, then work on+that one. Some fixes are flags; others need development work. The+[summary](#summary) at the end separates them.++## How a chunk spends its time++Experts live in host RAM. During prefill each MoE layer copies all of its+experts from host to GPU, for every chunk, however short (a large chunk routes+to nearly every expert anyway). For a chunk of `T` tokens+(`--max-prefill-length`, default 8192):++- **copy** depends on the expert bytes and the PCIe bandwidth+ (`ft bench bw` measures it), not on `T`. On PCIe 5.0, contiguous copies of+ 16 MiB or more already reach the full ~55.6 GB/s, and the per-layer expert+ copies are far larger, so copy time per chunk ≈ total expert bytes /+ bandwidth.+- **compute** grows with `T`, at a rate set by the model implementation and+ the quantization method (which kernels run, and how well they use the GPU).++The two run in one of two modes:++- **Serial**: each layer copies its experts, then computes.+ Chunk time = copy + compute.+- **Overlap** (the default): the copy of layer N+1 runs while layer N+ computes. Chunk time ≈ max(copy, compute).++## Step 1: make sure overlap is on++Overlap stages layers in two buffers carved out of the GPU expert cache, each+holding one whole layer of experts. If the cache cannot hold both, the server+logs at startup:++```+WARNING Disabling MoE prefill overlap: the slot cache cannot hold two whole layers of experts+```++and every prefill runs serial, paying copy + compute on every chunk. Look for+this warning in the startup log.++- Raise `--moe-cache-size` (or `--moe-cache-mem`) until the warning goes away.+ The extra slots come out of the VRAM left for KV and activations.+- Overlap is also off with `--disable-moe-prefill-overlap`, and when some+ layers are OS-locked instead of pinned, on hosts that cap pinned memory (see+ [models.md](../models.md#hosts-that-cap-pinned-memory-wsl)): locked layers+ can only be copied synchronously. A `--moe-cache-size` or `--moe-cache-rate`+ below two layers of experts fails at startup unless overlap is disabled.++Each buffer is sized for a whole layer at the widest expert format of any+layer, while most cache slots may hold a narrower format. A checkpoint that+mixes expert formats (for example unsloth's UD GGUF tiers) therefore needs+more than `2 x experts per layer` slots. Example: GLM-5.3-Flash UD-IQ4_XS has+288 experts per layer; its `ffn_down` experts are IQ4_XS in 39 MoE layers and+Q6_K in 3. The buffers are sized for Q6_K, so overlap needs about 856 slots+instead of 576. At 800 slots it is off.++## Step 2: find the bottleneck++With overlap on, raise `--max-prefill-length` and watch the prefill+`input throughput (token/s)` in the server log:++- It grows about in proportion to the chunk: **copy-bound**. The GPU waits+ for PCIe, and faster kernels will not help.+- It barely moves: **compute-bound**. The copy is already hidden, and a+ larger chunk will not help.++Throughput only hints at the bottleneck; to confirm it, profile a prefill+chunk with Nsight Systems (`nsys profile -t cuda,nvtx -o prefill ft serve ...`,+then send one long request). With overlap (and without+`--moe-prefill-hit-d2d`), each layer's expert copy is a `Memcpy HtoD` on its+own stream, next to the layer's kernels on the compute stream:++- Copy-bound: the HtoD copies run back to back and the compute stream idles+ between layers, waiting for the next layer's copy.+- Compute-bound: each copy finishes before the layer that needs it starts,+ and the copy stream idles.+- No overlap: each layer's copy runs on the compute stream (as a gather+ kernel or a memcpy) before the layer's kernels, never alongside them.++## Copy-bound: cut the PCIe time++1. **Raise the chunk size** (`--max-prefill-length`). The copy is the same+ per chunk, so a larger chunk gives compute more work to hide it behind,+ until compute catches up with the copy. A larger chunk also needs more+ activation memory, see [Activation memory](#activation-memory).+2. **Skip experts already on the GPU** (`--moe-prefill-hit-d2d`, CUDA >= 13).+ Experts already in the cache are copied on the GPU and only the misses+ cross PCIe. The gain grows with the share of experts the cache holds, so it+ matters most with a large cache (high `--moe-cache-rate`). It needs overlap+ and a cache larger than two layers; otherwise it falls back to the full+ copy and logs why once.+3. If the experts fit on the GPU, `--moe-strategy fused` removes the copy+ entirely.++## Compute-bound: speed up the forward pass++At about 8k tokens per chunk the large GEMMs already keep the GPU busy, so a+larger chunk adds little compute throughput. If it still does, the kernels+underuse the GPU at 8k, and that is a kernel problem, not a flag problem.++Compute-bound prefill needs kernel work (profile one chunk with Nsight+Systems first):++- **Quantized GEMMs on native tensor cores.** Check that the large quantized+ GEMMs use the GPU's native low-precision MMA (FP8, FP4) instead of+ dequantizing to bf16 or running a generic fallback.+- **Fused small ops.** Check that norms, activations, routing and similar+ small ops run as fused kernels rather than as chains of separate torch ops,+ each with its own launch and memory round trip.++A kernel speedup lowers compute, so it also raises the chunk size at which+compute hides the copy. After one, the chunk may need to grow again to stay+compute-bound.++## Activation memory++Prefill activations grow with the chunk size, but startup sizing does not+look at the chunk size: weights, expert cache and KV take their share of VRAM+(`--memory-ratio`, or a fixed `--num-tokens`), and prefill activations must+fit in what is left. A chunk that does not fit shows up as allocator warnings+during prefill:++```+expandable_segments: memory mapping failed with OOM+```++and in the worst case as an out-of-memory error.++- With flags: lower `--max-prefill-length`, or lower `--num-tokens` (or+ `--memory-ratio`) to leave more room for activations.+- With development: find what holds the peak. Record one chunk with+ `torch.cuda.memory._record_memory_history()`, save it with+ `torch.cuda.memory._dump_snapshot("prefill.pickle")`, and open the file as a+ flame graph at <https://pytorch.org/memory_viz>. Look for tensors copied+ more often than needed (for example a `.contiguous()` or `torch.cat` of a+ large activation that the next op does not require) and for large+ temporaries that outlive their use.++## Summary++| Situation | With flags | Needs development |+|---|---|---|+| Overlap off (startup warning) | raise `--moe-cache-size` / `--moe-cache-mem` | size the staging buffers for mixed-format checkpoints without the extra slots |+| Copy-bound | raise `--max-prefill-length`; `--moe-prefill-hit-d2d` with a large cache; `fused` if the experts fit | — |+| Compute-bound | — | native tensor-core quantized GEMMs, fused small ops |+| Larger chunk runs out of memory | lower `--max-prefill-length`, `--num-tokens` or `--memory-ratio` | memory snapshot, remove extra activation copies |