Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

Split serving: exactness and speed check tool, docs draft

Niko12213813a459c4ad
diff --git a/docs/SPLIT_SERVE.md b/docs/SPLIT_SERVE.md
new file mode 100644
index 000000000..55964f638
--- /dev/null
+++ b/docs/SPLIT_SERVE.md
@@ -0,0 +1,132 @@
+# Split serving: read the prompt on one card, make the answer on another
+
+> **Status: opt-in, first version, same make of cards on one PC.** Nothing changes unless you turn it on. It needs two cards
+> that can each run the whole model, and it only pays on some workloads. Read [When it helps](#when-it-helps) first.
+
+The idea, the hand-made setup and the first measurements are by @MarcoTool in issue #4000: a router in front of two
+engines, an RTX 3070 reading the prompt and two MI50 cards making the answer, with the session handed over through
+`/dev/shm`. This page describes the built-in version of the same idea.
+
+> **On this page:** [What it does](#what-it-does) · [Turning it on](#turning-it-on) · [When it helps](#when-it-helps) ·
+> [Measured](#measured) · [How it works](#how-it-works) · [What must match](#what-must-match) · [Limits](#limits) ·
+> [NVIDIA with AMD, and other PCs](#nvidia-with-amd-and-other-pcs)
+
+## What it does
+
+A normal two-card setup ([MULTI_GPU.md](MULTI_GPU.md)) cuts the model into layers and every card works on both stages. Split
+serving does not cut the model. It starts two engines, each on one card, each with the whole model:
+
+- the **prefill engine** reads the prompt and stops just before the first answer token;
+- the engine saves that session into RAM, the **decode engine** loads it, and makes the answer.
+
+The two engines share one copy of the experts in RAM (`--shared-expert-arena`, Linux). The client still talks to one
+OpenAI-compatible endpoint; the server hands the session over between the two.
+
+A prompt whose new part is short (a chat follow-up, an agent step) never leaves the decode engine: it is read in place, as
+without the option. Only a long new part goes to the prefill engine.
+
+## Turning it on
+
+In the run config (`strata-<model>.json`), next to the other keys:
+
+```json
+{
+ "split_serve": {"prefill_gpu": 0, "decode_gpu": 1}
+}
+```
+
+or `strata serve --split-serve 0,1` (prefill card first). The cards are numbered as `nvidia-smi` numbers them. The option
+replaces `gpu`; the config must not also name `replicas`, `--layer-split`, `--batch`/`--slots` or `--prompt-cache 0` (the
+server stops at start and says which).
+
+| Key | Default | Meaning |
+| --- | --- | --- |
+| `prefill_gpu`, `decode_gpu` | required | The two cards. Different cards. |
+| `min_prompt_tokens` | 2048 | A request is handed over when the decode engine would otherwise read at least this many new tokens. Below it the decode engine reads them in place. |
+| `dir` | `/dev/shm` if it has room, else the temp folder | Where the session file lives for the seconds of a hand-over. |
+| `shared_arena` | `true` | One copy of the experts in RAM for both engines (Linux, a file in `/dev/shm` when the experts fit there). `false`: each engine keeps its own copy (two copies of the experts in RAM). A file name: that file backs the arena (a file on disk works; it is written once). |
+| `prefill_args`, `decode_args` | none | Extra engine arguments for one role, for example a bigger expert cache for the decode engine. They must not change the [session settings](#what-must-match). |
+
+The startup line says what was chosen, including when the arena cannot be shared:
+
+```
+[strata] split serving: the prompt is read on card 0, the answer is made on card 1 (prompts with 2048+ new tokens are handed over); one shared expert arena in /dev/shm (32.0 GiB)
+```
+
+`/metrics` has a `split_serve` block: requests, hand-overs, fallbacks, bytes moved and the time spent saving and restoring.
+
+## When it helps
+
+Prefill needs matrix units and compute. Decode is limited by memory bandwidth and by how much of the model sits in VRAM. If one
+card is much better at the first and another at the second, each can do what it is best at. That was the case in #4000: an
+RTX 3070 reads a 32K prompt at about 1,600 tokens a second, the two MI50 cards read it at about 390 but decode at 40 tokens a
+second with every expert in VRAM. The hand-over costs 2 seconds at 32K.
+
+It does **not** pay when:
+
+- the two cards are alike: a layer split over both cards reads the prompt on both and keeps twice the experts in VRAM, which
+ split serving gives up (each engine has one card's worth of VRAM for experts);
+- the hand-over (the session file's size over RAM bandwidth, written and read) costs more than reading the new tokens on the
+ decode card would have;
+- the new part of each request is short, which is most chat and agent use: nothing is handed over.
+
+[Measured](#measured) shows the numbers on two identical cards.
+
+## Measured
+
+SPLIT-RESULTS-PLACEHOLDER
+
+## How it works
+
+1. The server sees a request with at least `min_prompt_tokens` new tokens for the decode engine (it tracks what the decode
+ engine's session holds).
+2. It sends the prompt to the prefill engine as `GEN 0 prefill=1 <ids>`. The engine reads the prompt as a request would, in
+ the same chunks, with the same checkpoints, up to the place where the first answer window starts, and stops. Its session
+ then holds all but the last prompt token. That is exactly the state a following turn resumes from, so the decode engine
+ needs no new code path.
+3. `SAVE <file>`: the engine's existing session file (see [DETAILS.md](DETAILS.md#session-files-disk)): the running state, the
+ K/V of every layer and the draft layer's K/V, the deepest checkpoint.
+4. `RESTORE <file>` on the decode engine, which checks the file against its own model files and settings.
+5. The request goes to the decode engine as usual. It finds the restored session as the start of the prompt, reads the last
+ token, and makes the answer. The file is deleted.
+
+If anything on the way fails (the prefill engine died, the file was refused, the card is out of memory), the decode engine reads
+the prompt itself, as without the option, and the log says why. A prefill engine that died is started again in the background.
+
+The figures the server reports for the request (prompt time, tokens read, tokens reused) count the prefill engine's work and
+the two file steps, so a client sees one request.
+
+## What must match
+
+The session file is bound to everything that changes what its bytes mean. Both engines start from one config, so these match;
+they would not if `prefill_args`/`decode_args` changed them:
+
+- the model files (sampled by size and first and last MiB) and the engine version;
+- the backend (CUDA or HIP): **no cross-backend restore**;
+- `--kv`, `--max-context`, `--kv-resident`, `--mtp-window`, `STRATA_KV_ROT`, the resolved rope settings, a loaded control vector;
+- the arithmetic switches that change the computed state (the engine lists them in the file's fingerprint).
+
+These may differ between the two engines: the expert cache size, `--prefill` chunk, sampling, drafting depth, the expert tier.
+That is the point of the two roles: the prefill engine can spend its VRAM on prompt buffers and the decode engine on experts.
+
+## Limits
+
+- One request at a time, as for a single engine. The prefill engine could read the next request's prompt while the decode engine
+ answers the current one; this version does not overlap them.
+- The decode engine holds one conversation. A hand-over replaces it (the engine's restore does not park the session it
+ replaces). A request that continues the conversation the decode engine holds is read in place.
+- No parallel slots (`--batch`), no layer split inside an engine, no pictures (a request with pictures goes to the decode
+ engine alone).
+- Needs RAM for the experts once (shared arena, Linux) or twice (otherwise), plus the session file while it exists:
+ about 20 MiB per 1,000 tokens at the default `--kv int8` on the Qwen3.8 model.
+- Exact token replay across a hand-over is as good as across any session restore: see [the check](#measured).
+
+## NVIDIA with AMD, and other PCs
+
+This version refuses nothing it cannot run: it runs on any two cards of one make that each hold the model. What the cases beyond
+it need:
+
+CROSS-VENDOR-PLACEHOLDER
+
+Credit: the idea, the hand-built router and the first measurements are @MarcoTool's (issue #4000, repository
+`MarcoTool/PDHyperMix-Strata-MI50`).
diff --git a/tools/split_serve_check.py b/tools/split_serve_check.py
new file mode 100644
index 000000000..810ed036f
--- /dev/null
+++ b/tools/split_serve_check.py
@@ -0,0 +1,301 @@
+#!/usr/bin/env python3
+"""Split serving (serve/split.py): the exactness check and the speed comparison, run on a test box with the real engine.
+
+ python3 tools/split_serve_check.py exact CONFIG.json OUT.json [--tokens N]
+ python3 tools/split_serve_check.py bench CONFIG.json OUT.json --arm layer|single|split [--reps 3]
+
+Run it from the repository root (it imports serve.server and serve.split). CONFIG.json:
+
+ {"exe": "/path/strata", "cwd": "/path/repo", "args": ["--serve", "--pack", ..., "--max-context", "40960", ...],
+ "env": {"STRATA_STATE_HASH": "1", ...}, "prompts": "/path/prompts", "dir": "/path/work",
+ "prefill_gpu": 0, "decode_gpu": 1, "arena": "/path/arena.bin" (optional), "min_prompt_tokens": 2048}
+
+`args` holds no --layer-split, --shared-expert-arena or card choice: the tool adds them for each arm.
+The prompts folder has pp_<N>.txt files of comma-separated token ids (the bench harness's).
+
+exact: a fresh engine reads a prompt and answers (the reference, twice, to show the engine's own run-to-run noise);
+then the prefill engine reads the same prompt, saves the session, a fresh decode engine restores it and answers. Compared:
+the state fingerprint (STRATA_STATE_HASH) of the prefill engine and of the decode engine right after the restore, then the
+answer's token ids and the fingerprint after the answer.
+
+bench: one arm per call (run it in its own job so the engines start fresh):
+ layer one engine over both cards (--layer-split auto), the normal two-card setup
+ single one engine on the decode card
+ split prefill engine on one card, decode engine on the other, through SplitEngine (the production path)
+Workloads: 8K and 32K prompt + 256 tokens, a 3-turn chat with short follow-ups, a 3-turn chat with long follow-ups.
+"""
+import argparse
+import copy
+import json
+import os
+import re
+import statistics
+import sys
+import threading
+import time
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+from serve import server as srv # noqa: E402
+from serve import split as split_mod # noqa: E402
+
+
+def read_ids(path):
+ return [int(x) for x in open(path).read().replace("\n", ",").split(",") if x.strip()]
+
+
+def mem_available_gib():
+ for line in open("/proc/meminfo"):
+ if line.startswith("MemAvailable"):
+ return int(line.split()[1]) / 1048576.0
+ return 0.0
+
+
+def make_engine(cfg, gpus, tag, extra_args=(), arena=None, min_free_gib=8.0):
+ free = mem_available_gib()
+ if free < min_free_gib:
+ raise SystemExit(f"only {free:.1f} GiB of RAM available (< {min_free_gib}): not starting an engine")
+ c = copy.deepcopy(cfg)
+ c["gpu"] = gpus
+ c["args"] = list(c["args"]) + list(extra_args) + (["--shared-expert-arena", arena] if arena else [])
+ c["log"] = os.path.join(cfg["dir"], f"engine-{tag}.log")
+ open(c["log"], "w").close()
+ env = srv.child_env(c)
+ env["STRATA_STATE_HASH"] = "1"
+ for k, v in (cfg.get("env") or {}).items():
+ env[str(k)] = str(v)
+ t0 = time.time()
+ e = srv.StrataEngine(c["exe"], srv.engine_args(c), cwd=c.get("cwd"), log=c["log"], env=env)
+ e.silence_s = 0
+ e.tag, e.load_s = tag, time.time() - t0
+ return e
+
+
+def log_tail(e, start):
+ try:
+ with open(e.log_path, "rb") as f:
+ f.seek(start)
+ return f.read().decode("utf-8", "replace")
+ except OSError:
+ return ""
+
+
+def log_size(e):
+ try:
+ return os.path.getsize(e.log_path)
+ except OSError:
+ return 0
+
+
+def state_hashes(text):
+ """The STATE_HASH lines of an engine log excerpt, as dicts."""
+ out = []
+ for m in re.finditer(r"STATE_HASH L=(\d+) (.*)", text):
+ d = {"L": int(m.group(1))}
+ d.update(dict(kv.split("=", 1) for kv in m.group(2).split() if "=" in kv))
+ out.append(d)
+ return out
+
+
+def run_gen(engine, ids, max_new):
+ """One request through engine.generate (a StrataEngine or a SplitEngine). -> dict of tokens and times."""
+ cancel = threading.Event()
+ toks, t_first = [], None
+ t0 = time.perf_counter()
+ for t in engine.generate(ids, max_new, {}, cancel):
+ if t is None:
+ continue
+ if t_first is None:
+ t_first = time.perf_counter()
+ toks.append(t)
+ t_end = time.perf_counter()
+ last = dict(engine.last or {})
+ dec = (len(toks) - 1) / (t_end - t_first) if t_first and len(toks) > 1 and t_end > t_first else None
+ return {"tokens": toks, "ttft_s": (t_first - t0) if t_first else None, "wall_s": t_end - t0,
+ "dec_tok_s": dec, "last": last}
+
+
+def close_all(*engines):
+ for e in engines:
+ try:
+ e.close()
+ except Exception as x: # noqa: BLE001
+ print("close:", x, flush=True)
+
+
+def cmd_exact(a, cfg):
+ ids = read_ids(os.path.join(cfg["prompts"], f"pp_{a.prompt}.txt"))[:a.prompt]
+ arena = cfg.get("arena")
+ P, Dg = cfg["prefill_gpu"], cfg["decode_gpu"]
+ res = {"prompt_tokens": len(ids), "new_tokens": a.tokens, "runs": {}}
+ # 1. the reference twice: a fresh engine on the decode card each time
+ for k in (1, 2):
+ e = make_engine(cfg, Dg, f"ref{k}", arena=arena)
+ pos = log_size(e)
+ r = run_gen(e, ids, a.tokens)
+ h = state_hashes(log_tail(e, pos))
+ res["runs"][f"ref{k}"] = {"tokens": r["tokens"], "hash_end": h[-1] if h else None, "last": r["last"],
+ "wall_s": r["wall_s"]}
+ print(f"ref{k}: {len(r['tokens'])} tokens, wall {r['wall_s']:.1f} s", flush=True)
+ close_all(e)
+ # 2. the split: prefill engine, save, fresh decode engine, restore, answer
+ pe = make_engine(cfg, P, "prefill", arena=arena)
+ pos = log_size(pe)
+ cancel = threading.Event()
+ for _ in pe.generate(ids, 0, {}, cancel, prefill_only=True):
+ pass
+ hp = state_hashes(log_tail(pe, pos))
+ path = os.path.join(cfg["dir"], "exact.session")
+ saved = pe.session_file("save", path)
+ print(f"prefill engine: held {saved['tokens']} tokens, saved {saved['bytes'] / 2**20:.0f} MiB in {saved['ms']:.0f} ms",
+ flush=True)
+ de = make_engine(cfg, Dg, "decode", arena=arena)
+ restored = de.session_file("restore", path)
+ print(f"decode engine: restored {restored['tokens']} tokens in {restored['ms']:.0f} ms", flush=True)
+ pos = log_size(de)
+ for _ in de.generate(ids, 0, {}, cancel, prefill_only=True): # nothing to read: only the state fingerprint
+ pass
+ hd = state_hashes(log_tail(de, pos))
+ pos = log_size(de)
+ r = run_gen(de, ids, a.tokens)
+ h = state_hashes(log_tail(de, pos))
+ res["runs"]["split"] = {"tokens": r["tokens"], "hash_end": h[-1] if h else None, "last": r["last"],
+ "wall_s": r["wall_s"], "hash_prefill_engine": hp[-1] if hp else None,
+ "hash_decode_after_restore": hd[-1] if hd else None, "saved": saved, "restored": restored}
+ print(f"split: {len(r['tokens'])} tokens, wall {r['wall_s']:.1f} s", flush=True)
+ close_all(pe, de)
+ try:
+ os.unlink(path)
+ except OSError:
+ pass
+ # verdicts
+ ref1, ref2, sp = (res["runs"][k] for k in ("ref1", "ref2", "split"))
+
+ def first_diff(x, y):
+ for i, (p, q) in enumerate(zip(x, y)):
+ if p != q:
+ return i
+ return None if len(x) == len(y) else min(len(x), len(y))
+ v = {
+ "restore_round_trip_state_equal": sp["hash_prefill_engine"] == sp["hash_decode_after_restore"] and
+ sp["hash_prefill_engine"] is not None,
+ "ref1_vs_ref2_first_diff": first_diff(ref1["tokens"], ref2["tokens"]),
+ "ref1_vs_split_first_diff": first_diff(ref1["tokens"], sp["tokens"]),
+ "ref2_vs_split_first_diff": first_diff(ref2["tokens"], sp["tokens"]),
+ "end_state_ref1_eq_ref2": ref1["hash_end"] == ref2["hash_end"],
+ "end_state_ref1_eq_split": ref1["hash_end"] == sp["hash_end"],
+ }
+ res["verdict"] = v
+ json.dump(res, open(a.out, "w"), indent=1)
+ print(json.dumps(v, indent=1), flush=True)
+ # the check passes when the split run equals the reference wherever the reference equals itself
+ same_ref = v["ref1_vs_ref2_first_diff"] is None
+ ok = v["restore_round_trip_state_equal"] and (
+ (v["ref1_vs_split_first_diff"] is None and v["end_state_ref1_eq_split"]) if same_ref
+ else v["ref1_vs_split_first_diff"] is None or v["ref2_vs_split_first_diff"] is None)
+ print("EXACT: PASS" if ok else "EXACT: FAIL", flush=True)
+ return 0 if ok else 1
+
+
+WORKLOADS = [
+ ("p8k", {"prompt": 8192, "gen": 256}),
+ ("p32k", {"prompt": 32768, "gen": 256}),
+ ("chat_short", {"prompt": 4096, "gen": 128, "turns": 3, "follow": 200}),
+ ("chat_long", {"prompt": 8192, "gen": 128, "turns": 3, "follow": 6000}),
+]
+
+
+def follow_ids(cfg, n, turn):
+ """n token ids for a follow-up message: a slice of the 16K prompt (different for each turn)."""
+ pool = read_ids(os.path.join(cfg["prompts"], "pp_16384.txt"))
+ start = 1000 + turn * 7000
+ return pool[start:start + n]
+
+
+def cmd_bench(a, cfg):
+ arm = a.arm
+ P, Dg = cfg["prefill_gpu"], cfg["decode_gpu"]
+ arena = cfg.get("arena")
+ t_load = time.time()
+ if arm == "layer":
+ eng = make_engine(cfg, [min(P, Dg), max(P, Dg)], "layer")
+ closers = [eng]
+ elif arm == "single":
+ eng = make_engine(cfg, Dg, "single")
+ closers = [eng]
+ elif arm == "split":
+ spec = split_mod.split_spec({"split_serve": {"prefill_gpu": P, "decode_gpu": Dg,
+ "min_prompt_tokens": cfg.get("min_prompt_tokens", 2048)}})
+ pe = make_engine(cfg, P, "prefill", extra_args=cfg.get("prefill_args", []), arena=arena)
+ de = make_engine(cfg, Dg, "decode", extra_args=cfg.get("decode_args", []), arena=arena)
+ eng = split_mod.SplitEngine(pe, de, min_tokens=spec["min_prompt_tokens"], work_dir=cfg["dir"], say=print)
+ closers = [pe, de]
+ else:
+ raise SystemExit("--arm layer|single|split")
+ load_s = time.time() - t_load
+ print(f"[{arm}] loaded in {load_s:.0f} s", flush=True)
+ rows = []
+ try:
+ warm = read_ids(os.path.join(cfg["prompts"], "dec_story.txt"))
+ for _ in range(3): # warm-up: graphs, the adaptive tier (discarded)
+ run_gen(eng, warm, 100)
+ if arm == "split":
+ run_gen(eng, read_ids(os.path.join(cfg["prompts"], "pp_4096.txt")), 16) # one hand-over: warms the file path
+ for rep in range(a.reps):
+ for name, w in WORKLOADS:
+ if a.only and name not in a.only:
+ continue
+ ids = read_ids(os.path.join(cfg["prompts"], f"pp_{w['prompt']}.txt"))[:w["prompt"]]
+ if "turns" not in w:
+ # a fresh conversation each time: a tiny unrelated request first, so no session is continued
+ run_gen(eng, warm[:40], 1)
+ r = run_gen(eng, ids, w["gen"])
+ turns = [r]
+ else:
+ run_gen(eng, warm[:40], 1)
+ turns, conv = [], list(ids)
+ for t in range(w["turns"]):
+ r = run_gen(eng, conv, w["gen"])
+ turns.append(r)
+ conv = conv + r["tokens"] + follow_ids(cfg, w["follow"], t)
+ row = {"arm": arm, "rep": rep, "workload": name, "load_s": round(load_s, 1), "turns": []}
+ for r in turns:
+ last = r["last"]
+ row["turns"].append({
+ "prompt": last.get("prompt_tokens"), "generated": len(r["tokens"]),
+ "ttft_s": round(r["ttft_s"], 3) if r["ttft_s"] else None, "wall_s": round(r["wall_s"], 3),
+ "dec_tok_s": round(r["dec_tok_s"], 2) if r["dec_tok_s"] else None,
+ "reused": last.get("reused"), "prompt_ms": last.get("prompt_ms"),
+ "split": {k: last[k] for k in last if k.startswith("split_")} or None,
+ "finish": last.get("finish"), "tokens": r["tokens"][:8]})
+ row["total_wall_s"] = round(sum(r["wall_s"] for r in turns), 3)
+ rows.append(row)
+ print(json.dumps({k: row[k] for k in ("arm", "rep", "workload", "total_wall_s")}), flush=True)
+ for t in row["turns"]:
+ print(" ", json.dumps({k: t[k] for k in ("prompt", "generated", "ttft_s", "wall_s", "dec_tok_s",
+ "reused", "split")}), flush=True)
+ finally:
+ out = {"arm": arm, "rows": rows, "cfg": {k: v for k, v in cfg.items() if k != "env"},
+ "split_view": eng.split_view() if hasattr(eng, "split_view") else None}
+ json.dump(out, open(a.out, "w"), indent=1)
+ close_all(*closers)
+ return 0
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("mode", choices=["exact", "bench"])
+ ap.add_argument("config")
+ ap.add_argument("out")
+ ap.add_argument("--tokens", type=int, default=64)
+ ap.add_argument("--prompt", type=int, default=4096)
+ ap.add_argument("--arm", default="split")
+ ap.add_argument("--reps", type=int, default=2)
+ ap.add_argument("--only", nargs="*", default=None)
+ a = ap.parse_args()
+ cfg = json.load(open(a.config))
+ os.makedirs(cfg["dir"], exist_ok=True)
+ return cmd_exact(a, cfg) if a.mode == "exact" else cmd_bench(a, cfg)
+
+
+if __name__ == "__main__":
+ sys.exit(main())