Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
Split serving: exactness and speed check tool, docs draft
Niko12213813a459c4ad
diff --git a/docs/SPLIT_SERVE.md b/docs/SPLIT_SERVE.mdnew file mode 100644index 000000000..55964f638--- /dev/null+++ b/docs/SPLIT_SERVE.md@@ -0,0 +1,132 @@+# Split serving: read the prompt on one card, make the answer on another++> **Status: opt-in, first version, same make of cards on one PC.** Nothing changes unless you turn it on. It needs two cards+> that can each run the whole model, and it only pays on some workloads. Read [When it helps](#when-it-helps) first.++The idea, the hand-made setup and the first measurements are by @MarcoTool in issue #4000: a router in front of two+engines, an RTX 3070 reading the prompt and two MI50 cards making the answer, with the session handed over through+`/dev/shm`. This page describes the built-in version of the same idea.++> **On this page:** [What it does](#what-it-does) · [Turning it on](#turning-it-on) · [When it helps](#when-it-helps) ·+> [Measured](#measured) · [How it works](#how-it-works) · [What must match](#what-must-match) · [Limits](#limits) ·+> [NVIDIA with AMD, and other PCs](#nvidia-with-amd-and-other-pcs)++## What it does++A normal two-card setup ([MULTI_GPU.md](MULTI_GPU.md)) cuts the model into layers and every card works on both stages. Split+serving does not cut the model. It starts two engines, each on one card, each with the whole model:++- the **prefill engine** reads the prompt and stops just before the first answer token;+- the engine saves that session into RAM, the **decode engine** loads it, and makes the answer.++The two engines share one copy of the experts in RAM (`--shared-expert-arena`, Linux). The client still talks to one+OpenAI-compatible endpoint; the server hands the session over between the two.++A prompt whose new part is short (a chat follow-up, an agent step) never leaves the decode engine: it is read in place, as+without the option. Only a long new part goes to the prefill engine.++## Turning it on++In the run config (`strata-<model>.json`), next to the other keys:++```json+{+ "split_serve": {"prefill_gpu": 0, "decode_gpu": 1}+}+```++or `strata serve --split-serve 0,1` (prefill card first). The cards are numbered as `nvidia-smi` numbers them. The option+replaces `gpu`; the config must not also name `replicas`, `--layer-split`, `--batch`/`--slots` or `--prompt-cache 0` (the+server stops at start and says which).++| Key | Default | Meaning |+| --- | --- | --- |+| `prefill_gpu`, `decode_gpu` | required | The two cards. Different cards. |+| `min_prompt_tokens` | 2048 | A request is handed over when the decode engine would otherwise read at least this many new tokens. Below it the decode engine reads them in place. |+| `dir` | `/dev/shm` if it has room, else the temp folder | Where the session file lives for the seconds of a hand-over. |+| `shared_arena` | `true` | One copy of the experts in RAM for both engines (Linux, a file in `/dev/shm` when the experts fit there). `false`: each engine keeps its own copy (two copies of the experts in RAM). A file name: that file backs the arena (a file on disk works; it is written once). |+| `prefill_args`, `decode_args` | none | Extra engine arguments for one role, for example a bigger expert cache for the decode engine. They must not change the [session settings](#what-must-match). |++The startup line says what was chosen, including when the arena cannot be shared:++```+[strata] split serving: the prompt is read on card 0, the answer is made on card 1 (prompts with 2048+ new tokens are handed over); one shared expert arena in /dev/shm (32.0 GiB)+```++`/metrics` has a `split_serve` block: requests, hand-overs, fallbacks, bytes moved and the time spent saving and restoring.++## When it helps++Prefill needs matrix units and compute. Decode is limited by memory bandwidth and by how much of the model sits in VRAM. If one+card is much better at the first and another at the second, each can do what it is best at. That was the case in #4000: an+RTX 3070 reads a 32K prompt at about 1,600 tokens a second, the two MI50 cards read it at about 390 but decode at 40 tokens a+second with every expert in VRAM. The hand-over costs 2 seconds at 32K.++It does **not** pay when:++- the two cards are alike: a layer split over both cards reads the prompt on both and keeps twice the experts in VRAM, which+ split serving gives up (each engine has one card's worth of VRAM for experts);+- the hand-over (the session file's size over RAM bandwidth, written and read) costs more than reading the new tokens on the+ decode card would have;+- the new part of each request is short, which is most chat and agent use: nothing is handed over.++[Measured](#measured) shows the numbers on two identical cards.++## Measured++SPLIT-RESULTS-PLACEHOLDER++## How it works++1. The server sees a request with at least `min_prompt_tokens` new tokens for the decode engine (it tracks what the decode+ engine's session holds).+2. It sends the prompt to the prefill engine as `GEN 0 prefill=1 <ids>`. The engine reads the prompt as a request would, in+ the same chunks, with the same checkpoints, up to the place where the first answer window starts, and stops. Its session+ then holds all but the last prompt token. That is exactly the state a following turn resumes from, so the decode engine+ needs no new code path.+3. `SAVE <file>`: the engine's existing session file (see [DETAILS.md](DETAILS.md#session-files-disk)): the running state, the+ K/V of every layer and the draft layer's K/V, the deepest checkpoint.+4. `RESTORE <file>` on the decode engine, which checks the file against its own model files and settings.+5. The request goes to the decode engine as usual. It finds the restored session as the start of the prompt, reads the last+ token, and makes the answer. The file is deleted.++If anything on the way fails (the prefill engine died, the file was refused, the card is out of memory), the decode engine reads+the prompt itself, as without the option, and the log says why. A prefill engine that died is started again in the background.++The figures the server reports for the request (prompt time, tokens read, tokens reused) count the prefill engine's work and+the two file steps, so a client sees one request.++## What must match++The session file is bound to everything that changes what its bytes mean. Both engines start from one config, so these match;+they would not if `prefill_args`/`decode_args` changed them:++- the model files (sampled by size and first and last MiB) and the engine version;+- the backend (CUDA or HIP): **no cross-backend restore**;+- `--kv`, `--max-context`, `--kv-resident`, `--mtp-window`, `STRATA_KV_ROT`, the resolved rope settings, a loaded control vector;+- the arithmetic switches that change the computed state (the engine lists them in the file's fingerprint).++These may differ between the two engines: the expert cache size, `--prefill` chunk, sampling, drafting depth, the expert tier.+That is the point of the two roles: the prefill engine can spend its VRAM on prompt buffers and the decode engine on experts.++## Limits++- One request at a time, as for a single engine. The prefill engine could read the next request's prompt while the decode engine+ answers the current one; this version does not overlap them.+- The decode engine holds one conversation. A hand-over replaces it (the engine's restore does not park the session it+ replaces). A request that continues the conversation the decode engine holds is read in place.+- No parallel slots (`--batch`), no layer split inside an engine, no pictures (a request with pictures goes to the decode+ engine alone).+- Needs RAM for the experts once (shared arena, Linux) or twice (otherwise), plus the session file while it exists:+ about 20 MiB per 1,000 tokens at the default `--kv int8` on the Qwen3.8 model.+- Exact token replay across a hand-over is as good as across any session restore: see [the check](#measured).++## NVIDIA with AMD, and other PCs++This version refuses nothing it cannot run: it runs on any two cards of one make that each hold the model. What the cases beyond+it need:++CROSS-VENDOR-PLACEHOLDER++Credit: the idea, the hand-built router and the first measurements are @MarcoTool's (issue #4000, repository+`MarcoTool/PDHyperMix-Strata-MI50`).diff --git a/tools/split_serve_check.py b/tools/split_serve_check.pynew file mode 100644index 000000000..810ed036f--- /dev/null+++ b/tools/split_serve_check.py@@ -0,0 +1,301 @@+#!/usr/bin/env python3+"""Split serving (serve/split.py): the exactness check and the speed comparison, run on a test box with the real engine.++ python3 tools/split_serve_check.py exact CONFIG.json OUT.json [--tokens N]+ python3 tools/split_serve_check.py bench CONFIG.json OUT.json --arm layer|single|split [--reps 3]++Run it from the repository root (it imports serve.server and serve.split). CONFIG.json:++ {"exe": "/path/strata", "cwd": "/path/repo", "args": ["--serve", "--pack", ..., "--max-context", "40960", ...],+ "env": {"STRATA_STATE_HASH": "1", ...}, "prompts": "/path/prompts", "dir": "/path/work",+ "prefill_gpu": 0, "decode_gpu": 1, "arena": "/path/arena.bin" (optional), "min_prompt_tokens": 2048}++`args` holds no --layer-split, --shared-expert-arena or card choice: the tool adds them for each arm.+The prompts folder has pp_<N>.txt files of comma-separated token ids (the bench harness's).++exact: a fresh engine reads a prompt and answers (the reference, twice, to show the engine's own run-to-run noise);+then the prefill engine reads the same prompt, saves the session, a fresh decode engine restores it and answers. Compared:+the state fingerprint (STRATA_STATE_HASH) of the prefill engine and of the decode engine right after the restore, then the+answer's token ids and the fingerprint after the answer.++bench: one arm per call (run it in its own job so the engines start fresh):+ layer one engine over both cards (--layer-split auto), the normal two-card setup+ single one engine on the decode card+ split prefill engine on one card, decode engine on the other, through SplitEngine (the production path)+Workloads: 8K and 32K prompt + 256 tokens, a 3-turn chat with short follow-ups, a 3-turn chat with long follow-ups.+"""+import argparse+import copy+import json+import os+import re+import statistics+import sys+import threading+import time++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+from serve import server as srv # noqa: E402+from serve import split as split_mod # noqa: E402+++def read_ids(path):+ return [int(x) for x in open(path).read().replace("\n", ",").split(",") if x.strip()]+++def mem_available_gib():+ for line in open("/proc/meminfo"):+ if line.startswith("MemAvailable"):+ return int(line.split()[1]) / 1048576.0+ return 0.0+++def make_engine(cfg, gpus, tag, extra_args=(), arena=None, min_free_gib=8.0):+ free = mem_available_gib()+ if free < min_free_gib:+ raise SystemExit(f"only {free:.1f} GiB of RAM available (< {min_free_gib}): not starting an engine")+ c = copy.deepcopy(cfg)+ c["gpu"] = gpus+ c["args"] = list(c["args"]) + list(extra_args) + (["--shared-expert-arena", arena] if arena else [])+ c["log"] = os.path.join(cfg["dir"], f"engine-{tag}.log")+ open(c["log"], "w").close()+ env = srv.child_env(c)+ env["STRATA_STATE_HASH"] = "1"+ for k, v in (cfg.get("env") or {}).items():+ env[str(k)] = str(v)+ t0 = time.time()+ e = srv.StrataEngine(c["exe"], srv.engine_args(c), cwd=c.get("cwd"), log=c["log"], env=env)+ e.silence_s = 0+ e.tag, e.load_s = tag, time.time() - t0+ return e+++def log_tail(e, start):+ try:+ with open(e.log_path, "rb") as f:+ f.seek(start)+ return f.read().decode("utf-8", "replace")+ except OSError:+ return ""+++def log_size(e):+ try:+ return os.path.getsize(e.log_path)+ except OSError:+ return 0+++def state_hashes(text):+ """The STATE_HASH lines of an engine log excerpt, as dicts."""+ out = []+ for m in re.finditer(r"STATE_HASH L=(\d+) (.*)", text):+ d = {"L": int(m.group(1))}+ d.update(dict(kv.split("=", 1) for kv in m.group(2).split() if "=" in kv))+ out.append(d)+ return out+++def run_gen(engine, ids, max_new):+ """One request through engine.generate (a StrataEngine or a SplitEngine). -> dict of tokens and times."""+ cancel = threading.Event()+ toks, t_first = [], None+ t0 = time.perf_counter()+ for t in engine.generate(ids, max_new, {}, cancel):+ if t is None:+ continue+ if t_first is None:+ t_first = time.perf_counter()+ toks.append(t)+ t_end = time.perf_counter()+ last = dict(engine.last or {})+ dec = (len(toks) - 1) / (t_end - t_first) if t_first and len(toks) > 1 and t_end > t_first else None+ return {"tokens": toks, "ttft_s": (t_first - t0) if t_first else None, "wall_s": t_end - t0,+ "dec_tok_s": dec, "last": last}+++def close_all(*engines):+ for e in engines:+ try:+ e.close()+ except Exception as x: # noqa: BLE001+ print("close:", x, flush=True)+++def cmd_exact(a, cfg):+ ids = read_ids(os.path.join(cfg["prompts"], f"pp_{a.prompt}.txt"))[:a.prompt]+ arena = cfg.get("arena")+ P, Dg = cfg["prefill_gpu"], cfg["decode_gpu"]+ res = {"prompt_tokens": len(ids), "new_tokens": a.tokens, "runs": {}}+ # 1. the reference twice: a fresh engine on the decode card each time+ for k in (1, 2):+ e = make_engine(cfg, Dg, f"ref{k}", arena=arena)+ pos = log_size(e)+ r = run_gen(e, ids, a.tokens)+ h = state_hashes(log_tail(e, pos))+ res["runs"][f"ref{k}"] = {"tokens": r["tokens"], "hash_end": h[-1] if h else None, "last": r["last"],+ "wall_s": r["wall_s"]}+ print(f"ref{k}: {len(r['tokens'])} tokens, wall {r['wall_s']:.1f} s", flush=True)+ close_all(e)+ # 2. the split: prefill engine, save, fresh decode engine, restore, answer+ pe = make_engine(cfg, P, "prefill", arena=arena)+ pos = log_size(pe)+ cancel = threading.Event()+ for _ in pe.generate(ids, 0, {}, cancel, prefill_only=True):+ pass+ hp = state_hashes(log_tail(pe, pos))+ path = os.path.join(cfg["dir"], "exact.session")+ saved = pe.session_file("save", path)+ print(f"prefill engine: held {saved['tokens']} tokens, saved {saved['bytes'] / 2**20:.0f} MiB in {saved['ms']:.0f} ms",+ flush=True)+ de = make_engine(cfg, Dg, "decode", arena=arena)+ restored = de.session_file("restore", path)+ print(f"decode engine: restored {restored['tokens']} tokens in {restored['ms']:.0f} ms", flush=True)+ pos = log_size(de)+ for _ in de.generate(ids, 0, {}, cancel, prefill_only=True): # nothing to read: only the state fingerprint+ pass+ hd = state_hashes(log_tail(de, pos))+ pos = log_size(de)+ r = run_gen(de, ids, a.tokens)+ h = state_hashes(log_tail(de, pos))+ res["runs"]["split"] = {"tokens": r["tokens"], "hash_end": h[-1] if h else None, "last": r["last"],+ "wall_s": r["wall_s"], "hash_prefill_engine": hp[-1] if hp else None,+ "hash_decode_after_restore": hd[-1] if hd else None, "saved": saved, "restored": restored}+ print(f"split: {len(r['tokens'])} tokens, wall {r['wall_s']:.1f} s", flush=True)+ close_all(pe, de)+ try:+ os.unlink(path)+ except OSError:+ pass+ # verdicts+ ref1, ref2, sp = (res["runs"][k] for k in ("ref1", "ref2", "split"))++ def first_diff(x, y):+ for i, (p, q) in enumerate(zip(x, y)):+ if p != q:+ return i+ return None if len(x) == len(y) else min(len(x), len(y))+ v = {+ "restore_round_trip_state_equal": sp["hash_prefill_engine"] == sp["hash_decode_after_restore"] and+ sp["hash_prefill_engine"] is not None,+ "ref1_vs_ref2_first_diff": first_diff(ref1["tokens"], ref2["tokens"]),+ "ref1_vs_split_first_diff": first_diff(ref1["tokens"], sp["tokens"]),+ "ref2_vs_split_first_diff": first_diff(ref2["tokens"], sp["tokens"]),+ "end_state_ref1_eq_ref2": ref1["hash_end"] == ref2["hash_end"],+ "end_state_ref1_eq_split": ref1["hash_end"] == sp["hash_end"],+ }+ res["verdict"] = v+ json.dump(res, open(a.out, "w"), indent=1)+ print(json.dumps(v, indent=1), flush=True)+ # the check passes when the split run equals the reference wherever the reference equals itself+ same_ref = v["ref1_vs_ref2_first_diff"] is None+ ok = v["restore_round_trip_state_equal"] and (+ (v["ref1_vs_split_first_diff"] is None and v["end_state_ref1_eq_split"]) if same_ref+ else v["ref1_vs_split_first_diff"] is None or v["ref2_vs_split_first_diff"] is None)+ print("EXACT: PASS" if ok else "EXACT: FAIL", flush=True)+ return 0 if ok else 1+++WORKLOADS = [+ ("p8k", {"prompt": 8192, "gen": 256}),+ ("p32k", {"prompt": 32768, "gen": 256}),+ ("chat_short", {"prompt": 4096, "gen": 128, "turns": 3, "follow": 200}),+ ("chat_long", {"prompt": 8192, "gen": 128, "turns": 3, "follow": 6000}),+]+++def follow_ids(cfg, n, turn):+ """n token ids for a follow-up message: a slice of the 16K prompt (different for each turn)."""+ pool = read_ids(os.path.join(cfg["prompts"], "pp_16384.txt"))+ start = 1000 + turn * 7000+ return pool[start:start + n]+++def cmd_bench(a, cfg):+ arm = a.arm+ P, Dg = cfg["prefill_gpu"], cfg["decode_gpu"]+ arena = cfg.get("arena")+ t_load = time.time()+ if arm == "layer":+ eng = make_engine(cfg, [min(P, Dg), max(P, Dg)], "layer")+ closers = [eng]+ elif arm == "single":+ eng = make_engine(cfg, Dg, "single")+ closers = [eng]+ elif arm == "split":+ spec = split_mod.split_spec({"split_serve": {"prefill_gpu": P, "decode_gpu": Dg,+ "min_prompt_tokens": cfg.get("min_prompt_tokens", 2048)}})+ pe = make_engine(cfg, P, "prefill", extra_args=cfg.get("prefill_args", []), arena=arena)+ de = make_engine(cfg, Dg, "decode", extra_args=cfg.get("decode_args", []), arena=arena)+ eng = split_mod.SplitEngine(pe, de, min_tokens=spec["min_prompt_tokens"], work_dir=cfg["dir"], say=print)+ closers = [pe, de]+ else:+ raise SystemExit("--arm layer|single|split")+ load_s = time.time() - t_load+ print(f"[{arm}] loaded in {load_s:.0f} s", flush=True)+ rows = []+ try:+ warm = read_ids(os.path.join(cfg["prompts"], "dec_story.txt"))+ for _ in range(3): # warm-up: graphs, the adaptive tier (discarded)+ run_gen(eng, warm, 100)+ if arm == "split":+ run_gen(eng, read_ids(os.path.join(cfg["prompts"], "pp_4096.txt")), 16) # one hand-over: warms the file path+ for rep in range(a.reps):+ for name, w in WORKLOADS:+ if a.only and name not in a.only:+ continue+ ids = read_ids(os.path.join(cfg["prompts"], f"pp_{w['prompt']}.txt"))[:w["prompt"]]+ if "turns" not in w:+ # a fresh conversation each time: a tiny unrelated request first, so no session is continued+ run_gen(eng, warm[:40], 1)+ r = run_gen(eng, ids, w["gen"])+ turns = [r]+ else:+ run_gen(eng, warm[:40], 1)+ turns, conv = [], list(ids)+ for t in range(w["turns"]):+ r = run_gen(eng, conv, w["gen"])+ turns.append(r)+ conv = conv + r["tokens"] + follow_ids(cfg, w["follow"], t)+ row = {"arm": arm, "rep": rep, "workload": name, "load_s": round(load_s, 1), "turns": []}+ for r in turns:+ last = r["last"]+ row["turns"].append({+ "prompt": last.get("prompt_tokens"), "generated": len(r["tokens"]),+ "ttft_s": round(r["ttft_s"], 3) if r["ttft_s"] else None, "wall_s": round(r["wall_s"], 3),+ "dec_tok_s": round(r["dec_tok_s"], 2) if r["dec_tok_s"] else None,+ "reused": last.get("reused"), "prompt_ms": last.get("prompt_ms"),+ "split": {k: last[k] for k in last if k.startswith("split_")} or None,+ "finish": last.get("finish"), "tokens": r["tokens"][:8]})+ row["total_wall_s"] = round(sum(r["wall_s"] for r in turns), 3)+ rows.append(row)+ print(json.dumps({k: row[k] for k in ("arm", "rep", "workload", "total_wall_s")}), flush=True)+ for t in row["turns"]:+ print(" ", json.dumps({k: t[k] for k in ("prompt", "generated", "ttft_s", "wall_s", "dec_tok_s",+ "reused", "split")}), flush=True)+ finally:+ out = {"arm": arm, "rows": rows, "cfg": {k: v for k, v in cfg.items() if k != "env"},+ "split_view": eng.split_view() if hasattr(eng, "split_view") else None}+ json.dump(out, open(a.out, "w"), indent=1)+ close_all(*closers)+ return 0+++def main():+ ap = argparse.ArgumentParser()+ ap.add_argument("mode", choices=["exact", "bench"])+ ap.add_argument("config")+ ap.add_argument("out")+ ap.add_argument("--tokens", type=int, default=64)+ ap.add_argument("--prompt", type=int, default=4096)+ ap.add_argument("--arm", default="split")+ ap.add_argument("--reps", type=int, default=2)+ ap.add_argument("--only", nargs="*", default=None)+ a = ap.parse_args()+ cfg = json.load(open(a.config))+ os.makedirs(cfg["dir"], exist_ok=True)+ return cmd_exact(a, cfg) if a.mode == "exact" else cmd_bench(a, cfg)+++if __name__ == "__main__":+ sys.exit(main())