Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Split serving: docs with the measured results, check tool no longer hashes the state in speed runs

Niko12213480bf2b8c6e
diff --git a/docs/SPLIT_SERVE.md b/docs/SPLIT_SERVE.md
index 55964f638..9eaee8b7d 100644
--- a/docs/SPLIT_SERVE.md
+++ b/docs/SPLIT_SERVE.md
@@ -1,7 +1,8 @@
# Split serving: read the prompt on one card, make the answer on another
-> **Status: opt-in, first version, same make of cards on one PC.** Nothing changes unless you turn it on. It needs two cards
-> that can each run the whole model, and it only pays on some workloads. Read [When it helps](#when-it-helps) first.
+> **Status: opt-in, first version, same make of cards on one PC. Not recommended on two alike cards: measured slower than the layer
+> split on every workload we tried ([Measured](#measured)).** Nothing changes unless you turn it on. It needs two cards that can
+> each run the whole model, and it can only pay when one card is clearly better at reading prompts and another at making answers.
The idea, the hand-made setup and the first measurements are by @MarcoTool in issue #4000: a router in front of two
engines, an RTX 3070 reading the prompt and two MI50 cards making the answer, with the session handed over through
@@ -35,7 +36,7 @@ In the run config (`strata-<model>.json`), next to the other keys:
}
```
-or `strata serve --split-serve 0,1` (prefill card first). The cards are numbered as `nvidia-smi` numbers them. The option
+or `python serve/server.py --split-serve 0,1` (prefill card first). The cards are numbered as `nvidia-smi` numbers them. The option
replaces `gpu`; the config must not also name `replicas`, `--layer-split`, `--batch`/`--slots` or `--prompt-cache 0` (the
server stops at start and says which).
@@ -64,8 +65,8 @@ second with every expert in VRAM. The hand-over costs 2 seconds at 32K.
It does **not** pay when:
-- the two cards are alike: a layer split over both cards reads the prompt on both and keeps twice the experts in VRAM, which
- split serving gives up (each engine has one card's worth of VRAM for experts);
+- the two cards are alike. A layer split over both cards reads a long prompt faster than one card and decodes faster than one card.
+ Split serving gives both up, and each engine has only one card's VRAM for experts. On two RTX A6000 it was 4% to 48% slower;
- the hand-over (the session file's size over RAM bandwidth, written and read) costs more than reading the new tokens on the
decode card would have;
- the new part of each request is short, which is most chat and agent use: nothing is handed over.
@@ -74,7 +75,59 @@ It does **not** pay when:
## Measured
-SPLIT-RESULTS-PLACEHOLDER
+Two identical cards, so this is the case where split serving has the least to offer: 2x RTX A6000 48 GB (a lent box, Linux,
+CUDA 13.1), Qwen3.8-Flash-Next IQ3_S, `--kv int8`, MTP drafting, default engine flags, greedy decoding. Three arms, each started
+fresh for every round (3 rounds, 2 repetitions each, so 6 values per cell; every request is a cold prompt, no cache hit):
+
+- **layer**: one engine over both cards (`--layer-split auto`), the normal two-card setup;
+- **single**: one engine on one card;
+- **split**: the prefill engine on one card, the decode engine on the other, through this option.
+
+Wall time of the whole request, median of 6 (a prompt of the given length and 256 answer tokens; the chats are 3 turns, 128 answer
+tokens each, follow-ups of 200 or 6,000 tokens, no turn markers in the token stream):
+
+| Workload | layer | single | split | split / layer | split faster than layer |
+| --- | --- | --- | --- | --- | --- |
+| 8K prompt + 256 | 4.91 s | 5.27 s | 5.69 s | 1.16 | 0 of 6 |
+| 32K prompt + 256 | 9.82 s | 13.64 s | 14.48 s | 1.48 | 0 of 6 |
+| chat, 4K + two 200-token follow-ups | 4.75 s | 5.27 s | 5.99 s | 1.30 | 0 of 6 |
+| chat, 8K + two 6,000-token follow-ups | 12.09 s | 13.43 s | 12.58 s | 1.04 | 0 of 6 |
+
+Where the time goes:
+
+| | layer | single | split |
+| --- | --- | --- | --- |
+| Prompt reading, 8K (to first token) | 3.10 s | 3.25 s | 3.68 s (3.23 s on the prefill card + 0.41 s hand-over) |
+| Prompt reading, 32K | 7.89 s | 11.53 s | 12.34 s (11.54 s + 0.74 s) |
+| Decode, tokens per second at 8K / 32K | 141 / 131 | 126 / 120 | 128 / 117 |
+
+- **The hand-over is small but not free:** 0.20 s to save and 0.19 s to restore a 345 MiB session at 8K, 0.35 s and 0.39 s for 702 MiB
+ at 32K. The file is about 225 MiB plus 14.7 MiB for every 1,000 tokens (285 MiB at 4K, 464 MiB at 16K). That is 7% of an 8K request and
+ 5% of a 32K one.
+- **Alike cards give it nothing to win with.** The prefill card reads at the speed of one card (about 2,500 to 2,900 tokens a
+ second) and the decode card decodes at the speed of one card. A layer split reads a 32K prompt in 32% less time than one card,
+ because the pipeline keeps both cards busy, and decodes 9% to 12% faster. It wins every request here.
+- **Short follow-ups are not handed over** (they are under `min_prompt_tokens`): the decode card reads them in place, as one engine would.
+- **Long follow-ups are handed over every time** (the second chat). The prefill engine does not hold the answer the decode engine just
+ made, so it reads the answer and the follow-up again (2.5 s for 6,128 tokens) and the session moves again. That chat is the closest
+ to the layer split (12.58 s against 12.09 s). Nothing runs at the same time in this version, so the two cards still take turns.
+
+**Exactness.** A fresh engine and the split path give the same answer, token for token, and the same session state:
+
+| Check | Model, cards | Prompt | New tokens | Result |
+| --- | --- | --- | --- | --- |
+| two fresh single engines, then prefill engine + session file + fresh decode engine | Q2_0, 2x RTX A4000 | 4,096 | 64 | tokens equal, state fingerprints equal |
+| the same | IQ3_S, 2x RTX A6000 | 4,096 | 64 | tokens equal, state fingerprints equal |
+| the same | IQ3_S, 2x RTX A6000 | 16,384 | 128 | tokens equal, state fingerprints equal |
+
+"State fingerprint" is the engine's `STRATA_STATE_HASH`: the whole session (running state, K/V of every layer, the draft layer's K/V) hashed after
+the request. The fingerprint of the prefill engine's session also equals the decode engine's right after the restore. The check ran with
+`--adapt-swaps 0 --pcie-frac 0 STRATA_IQ_MT_MIN=1`, the settings under which two fresh single engines already give the same answer
+(without them the engine is not always deterministic from run to run, with or without this option). The tool is `tools/split_serve_check.py exact`.
+
+On two alike cards, split serving does not pay for one request at a time. It can pay when one card is clearly better at reading
+prompts and another at making answers, as in #4000. The gain there is the prompt time the weaker card would have needed, minus the
+hand-over (0.4 s at 8K and 0.7 s at 32K on this box). We have not measured such a pair.
## How it works
@@ -118,15 +171,48 @@ That is the point of the two roles: the prefill engine can spend its VRAM on pro
- No parallel slots (`--batch`), no layer split inside an engine, no pictures (a request with pictures goes to the decode
engine alone).
- Needs RAM for the experts once (shared arena, Linux) or twice (otherwise), plus the session file while it exists:
- about 20 MiB per 1,000 tokens at the default `--kv int8` on the Qwen3.8 model.
-- Exact token replay across a hand-over is as good as across any session restore: see [the check](#measured).
+ about 225 MiB plus 14.7 MiB per 1,000 tokens at `--kv int8` on the Qwen3.8 model.
+- Token for token the same answer as one engine, in the checks listed under [Measured](#measured).
## NVIDIA with AMD, and other PCs
-This version refuses nothing it cannot run: it runs on any two cards of one make that each hold the model. What the cases beyond
-it need:
-
-CROSS-VENDOR-PLACEHOLDER
+This version runs on any two cards of one make that each hold the model (the engine drives one make of card, and a session file is
+bound to its backend). It was tested on NVIDIA only. What the cases beyond it need, from reading the code and from #4000, none of it
+built or measured here:
+
+**NVIDIA with AMD on one PC** (the setup in #4000):
+
+- *The session file must be allowed across backends.* Today the file's fingerprint contains the backend (`cuda` or `hip`) and the
+ arithmetic switches that differ per card family, so a restore is refused. A cross-backend mode has to compare only what defines the
+ bytes: model files, engine version, `--kv`, `--max-context`, `--kv-resident`, `--mtp-window`, the rope settings, `STRATA_KV_ROT`. The K/V,
+ running-state and draft-layer layouts come from one source tree, but nobody has checked that the bytes are interchangeable on both
+ backends.
+- *A byte check is no longer possible.* The two backends do not compute the same bits, so the proof is not a state fingerprint. It
+ would be a teacher-forced KL of the answer after a hand-over against the same backend's own continuation, with the noise floor of the
+ decode engine against itself, plus needle and task checks, as for any change of arithmetic.
+- *Two engine builds.* One CUDA and one HIP executable of the same version, so the config needs an executable and a card per role
+ (`prefill_exe`, `decode_exe`), the card variable per backend (`CUDA_VISIBLE_DEVICES`, `HIP_VISIBLE_DEVICES`), and a start check that both
+ report the same version.
+- *The shared expert arena works across makes* (#4000 does it with `--shared-expert-arena` in `/dev/shm`): it holds the pack's bytes and
+ each engine pins them with its own runtime. It needs the experts to fit `/dev/shm`, or a file on disk, as for the same-make case.
+- *Roles chosen from measurement.* Setup and `--calibrate` would read each card's prompt speed and decode speed and suggest which
+ card reads and which answers, and say when the pair is not worth splitting. Without that the owner of the PC has to know.
+- *A test machine with both makes.* We have none in 0.2.7.
+
+**Another PC** (prefill on one machine, decode on another):
+
+- The session file is 225 MiB plus 14.7 MiB per 1,000 tokens (about 2 GiB at 128K). A 10 Gb/s link moves that in 2 to 3 seconds; a
+ gigabit link in 20.
+- Each PC loads its own copy of the experts, so there is no shared arena. Both need the same model files and engine version; the
+ fingerprint samples the files' size and first and last MiB, not their path.
+- The engines read a session file from a local path and trust it ("restore only files this engine wrote"). A network hand-over needs a
+ transport with authentication and an upload/download call on the server, which does not exist.
+- Requests would have to be routed to the PC that holds the conversation, which is the replica router's job already.
+
+**Several clients at once** (not built, not measured): the prefill card could read the next request's prompt while the decode card
+answers the current one. That would raise the throughput of a busy server. It would not make one request faster than the layer split on
+alike cards. This version runs one request at a time. Overlapping needs the server to take two requests at once (a prefill queue and a
+decode queue), and the decode engine to take a restore between two of its own requests, which the session file already allows.
Credit: the idea, the hand-built router and the first measurements are @MarcoTool's (issue #4000, repository
`MarcoTool/PDHyperMix-Strata-MI50`).
diff --git a/tools/split_serve_check.py b/tools/split_serve_check.py
index e2ac1166c..f441bdb0b 100644
--- a/tools/split_serve_check.py
+++ b/tools/split_serve_check.py
@@ -7,7 +7,7 @@
Run it from the repository root (it imports serve.server and serve.split). CONFIG.json:
{"exe": "/path/strata", "cwd": "/path/repo", "args": ["--serve", "--pack", ..., "--max-context", "40960", ...],
- "env": {"STRATA_STATE_HASH": "1", ...}, "prompts": "/path/prompts", "dir": "/path/work",
+ "env": {...}, "prompts": "/path/prompts", "dir": "/path/work",
"prefill_gpu": 0, "decode_gpu": 1, "arena": "/path/arena.bin" (optional), "min_prompt_tokens": 2048}
`args` holds no --layer-split, --shared-expert-arena or card choice: the tool adds them for each arm.
@@ -62,7 +62,7 @@ def mem_available_gib():
return 0.0
-def make_engine(cfg, gpus, tag, extra_args=(), arena=None, min_free_gib=8.0):
+def make_engine(cfg, gpus, tag, extra_args=(), arena=None, min_free_gib=8.0, state_hash=False):
free = mem_available_gib()
if free < min_free_gib:
raise SystemExit(f"only {free:.1f} GiB of RAM available (< {min_free_gib}): not starting an engine")
@@ -72,7 +72,8 @@ def make_engine(cfg, gpus, tag, extra_args=(), arena=None, min_free_gib=8.0):
c["log"] = os.path.join(cfg["dir"], f"engine-{tag}.log")
open(c["log"], "w").close()
env = srv.child_env(c)
- env["STRATA_STATE_HASH"] = "1"
+ if state_hash: # the fingerprint after every request costs time proportional to the prompt: exact mode only
+ env["STRATA_STATE_HASH"] = "1"
for k, v in (cfg.get("env") or {}).items():
env[str(k)] = str(v)
t0 = time.time()
@@ -141,7 +142,7 @@ def cmd_exact(a, cfg):
res = {"prompt_tokens": len(ids), "new_tokens": a.tokens, "runs": {}}
# 1. the reference twice: a fresh engine on the decode card each time
for k in (1, 2):
- e = make_engine(cfg, Dg, f"ref{k}", arena=arena)
+ e = make_engine(cfg, Dg, f"ref{k}", arena=arena, state_hash=True)
pos = log_size(e)
r = run_gen(e, ids, a.tokens)
h = state_hashes(log_tail(e, pos))
@@ -150,7 +151,7 @@ def cmd_exact(a, cfg):
print(f"ref{k}: {len(r['tokens'])} tokens, wall {r['wall_s']:.1f} s", flush=True)
close_all(e)
# 2. the split: prefill engine, save, fresh decode engine, restore, answer
- pe = make_engine(cfg, P, "prefill", arena=arena)
+ pe = make_engine(cfg, P, "prefill", arena=arena, state_hash=True)
pos = log_size(pe)
cancel = threading.Event()
for _ in pe.generate(ids, 0, {}, cancel, prefill_only=True):
@@ -160,7 +161,7 @@ def cmd_exact(a, cfg):
saved = pe.session_file("save", path)
print(f"prefill engine: held {saved['tokens']} tokens, saved {saved['bytes'] / 2**20:.0f} MiB in {saved['ms']:.0f} ms",
flush=True)
- de = make_engine(cfg, Dg, "decode", arena=arena)
+ de = make_engine(cfg, Dg, "decode", arena=arena, state_hash=True)
restored = de.session_file("restore", path)
print(f"decode engine: restored {restored['tokens']} tokens in {restored['ms']:.0f} ms", flush=True)
pos = log_size(de)